Codehelp blog

Steve's docs on keys on a USB stick needed a few tweaks for my own systems - first thing I noticed was that I had /dev/loop0 instead of /dev/loop/0. As my configuration needs to use an SDcard instead of a USB stick or pen, I've changed the references from 'pen' to 'card'.
SDcards on my systems come out on /dev/mmcblk0 with partitions having a 'p[0-9]' suffix (p1, p2 etc.) so the DEVBASE setting needed a tweak. (This line needs to be modified for each user case anyway because the size of the partition will vary so the '62390272' value needs to be modified.)
DEVBASE=`fdisk -l /dev/mmcblk0 2>/dev/null | grep ^Disk.*62390272\ bytes | sed 's?^.*/d?/d?g;s?:.*$??g'`
A few other notes for those using Sid instead of Etch:

1. kernel module needed: loop-aes-modules-2.6.26-1 - getting the modules to load requires a reboot.


2. loop-aes-utils the actual package that provides the diverted mount binaries, instead of just "loop-aes"

Barring a few typos I made in transcribing, the rest of the documentation works fine.

Alternative keys

I have two keys, one is intended to remain on the laptop and has comments to that effect (0xA897FD02). The other key, (0x28BCB3E3), is my Debian upload key so needs to be protected a little more than the ordinary laptop key (which I'll use when signing email on the laptop).
'debsign' does not have a way to specify a different --homedir or secret keyring or any other option that would allow me to have an alternative gpg configuration (maybe it should?). So I've copied debsign into my home scripts directory to create a slightly modified version and then I followed Steve's advice for ~/.gnupg/ to create a duplicate directory with the same permissions and a much reduced public keyring file (just exported my own keys). The secring.gpg file for this alternative homedir is the on the SD card with the autofs symlink (so the only file on the encrypted SDCard is the secret keyring).
Then it is a simple case of manually calling debsign (e.g. after using debuild -uc -us):
~/scripts/debsign -k0x28bcb3e3 /path/to/.changes

+CARD=" --no-default-keyring --homedir=/home/neil/.gcard "

...

-                   $signcommand --local-user "$2" --clearsign \

+                   $signcommand $CARD --local-user "$2" --clearsign \

...

-                   $signcommand --local-user "$2" --clearsign \

+                   $signcommand $CARD --local-user "$2" --clearsign \

...

-       $signcommand -u "$2" +clearsig=on -fast < "$1" > "$ASCII_SIGNED_FILE"

+       $signcommand $CARD -u "$2" +clearsig=on -fast < "$1" > "$ASCII_SIGNED_FILE"

po4a

- or PO for anything. PO is the de facto standard for translations and po4a tries to help maintainers prepare packages for translation.

1. I18n (internationalisation) is making a package accessible for translators and is an upstream/maintainer action. i18n is done once for all translation support and updated when the content changes (typically during the build).


2. l10n (localisation) is using i18n support to create an individual translation for a specific locale. l10n is done many times per package, with individual translations being updated each time the i18n content changes. (Tdebs will support this area but are a separate topic from po4a.) PO files that exist when the POT is updated are also updated, tagging modified strings as "fuzzy".

The natural burden, therefore, falls within l10n but without i18n, l10n is all but impossible (it means having to do the i18n "outside" the package).
I'm using po4a and I've been doing various translation-support stuff now for a few years, so I'm going to add po4a into my blog entries as a series to help maintainers make their own packages i18n aware.
This is particularly important for:

1. all Debian native packages, particularly scripted packages. My example will be based on perlpod documentation often embedded wthin the perl scripts themselves as well as DocBook XML for manpage generation. po4a supports many other formats.


2. All manpages (or other documentation) created by Debian maintainers

To make things easier, I'll be creating a new category in my blog ("i18n") for such entries. This is helpful for me too because the first time I used po4a (current versions of emdebian-tools) I basically got things wrong and made more work for myself than was necessary. This entry primarily concerns providing i18n support from manpage sources and perl scripts. i18n support in compiled packages is a separate issue. The fixes described here will be included in the next version of emdebian-tools (which will be uploaded to Emdebian but not into Debian until after Lenny).

Terminology

• PO - (Portable Object) files contain the POSIX C locale strings abstracted from the "master document" which in many cases is the source file in the relevant package(s), alongside the translated version of that string (if any). Multiple master documents are typically combined into a single collection of translatable strings. PO files are translated into individual locales, one PO file per locale.


• POT - PO Template - the definitive collection of translatable strings from which all PO files are prepared and updated. The POT file is never translated directly, it is maintained, updated and packaged by the maintainer.


• gettext - the binary application that handles the translatable strings. gettext abstracts the strings out of the source files during the package build and inserts the translated strings back into the text actually displayed to the user at runtime. If no suitable PO file exists, the user gets the untranslated string, exactly as it appears in the POT file. If a suitable PO file does exist but the strings have been modified since translation, gettext drops "fuzzy" translations that do not match precisely and the user gets a "mixed" translation containing some translated strings and some untranslated. Most translators try to maintain a 100% translation because the ugliness of the "mixed" output is a partial incentive to keep up with the updates.

Using po4a

po4a needs a config file where the maintainer keeps all metadata relating to the master documents and the relevant formats. po4a then converts the master documents into a collection of strings for translation and puts those strings into a POT file (a PO template file). The maintainer packages the POT file (and optionally sends the POT file to relevant mailing lists seeking translations to be returned as PO files) and keeps the POT file up to date during the build process. With a config file (e.g. called po4a.config), all the maintainer needs to do is call:

po4a po4a.config

po4a then parses the list of files defined in the config file, generates the lists of strings, collates the lists and identifies where the same string is used in multiple files and writes out a POT file containing references to where the strings are used and the unique strings.

po4a config file

Comments start with '#' in the usual way. Instructions are enclosed in square brackets.
po4a_paths defines where the POT file will be put (it can be extended later but I'm keeping this simple). The name of the POT file is important for compiled packages but can just be the name of the source package for scripted packages and/or manpages.

[po4a_paths] po/emdebian-tools.pot

type identifies *each* master document to be processed by po4a. The syntax is well suited to being generated from some other list within the package but I'll give just a snippet of the final output here:

[type: xml] doc/xml/em_autobuild.1.xml

[type: xml] doc/xml/embug.1.xml

[type: pod] emtargetcmp

[type: pod] emdebcheck

That's it. The complexity of my current script was entirely unnecessary.

Also, po4a is more than capable of handling multiple document types within the same source package and the same POT file which is something that I had missed so far. One more hack that can be removed.

All I do now is use xmllint to "clean-up" the DocBook XML and pod2html and some XSLT to generate HTML documentation for the website. The loops to do those tasks already contain the list of files to be processed by po4a, so a few echo commands is all that is needed to create po4a.config at build time and run po4a to generate a combined POT file for all strings in the package.

#!/bin/sh

CONFIG="po4a.config"

echo "[po4a_paths] po/emdebian-tools.pot" > $CONFIG

for file in `ls doc/xml/*.xml`; do

	xmllint --format "$file" --output "$file"

	echo "[type:xml] $file" >> $CONFIG

done

for file in emdebcheck emtargetcmp emprunecross Emdebian/Tools.pm \

 em_installtdeb emrecent emcache; do

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2html -outfile html/EmdebianTools.html -title $file < ../$file

	else

		pod2html -outfile html/$file.html -title $file < ../$file

	fi

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2man ../$file > man/Emdebian::Tools.3

	else

		pod2man ../$file > man/$file.1

	fi

	echo "[type: pod] $file" >> $CONFIG

done

po4a $CONFIG

First complete day after a full day travelling and I'm just about over the jet lag. Progress so far:

• Restored the previous Gtk+ build configuration - without directfb and therefore without the udeb. Persistent problems resolving symbols in libcairo-directfb (in libcairo.so) due to the presence of libcairo itself (also libcairo.so) mean that the directfb code simply cannot cross-build at the moment. A possible solution might be to rename the library in libcairo-directfb-$arch-cross but that isn't necessarily correct.


• Further reductions in the number of packages that fail to cross-build in a clean cross-building chroot. Down to just 3 that do not cross-build (added code to ignore obsolete builds like gcc-4.2 which have been superceded in Debian). A few network issues have delayed uploads and sync with www.emdebian.org but those should be resolved in time.


• Fixed a loose wire in the balloon board that was causing intermittent loss of serial connection - thanks to Phil Hands and Wookey.


• Resolved the chpasswd problem in busybox thanks to a fix upstream by Bernhard Fischer - removing the last hurdle to clean root filesystem installation in Emdebian. I'm currently working on busybox 1.11.1 to both test the fix and prepare the Emdebian patches for the pending upgrade in Debian. I do have an updated Emdebian build for 1.11.1 which I'll be testing later.


• Looked at the possibility of including nbd into Emdebian - it cross-builds OK but the maintainer scripts rely on bash which is simply not available in Emdebian. Wouter is working on a fix, hopefully ready sometime after Lenny.


• Taken a look at freeswitch (not in Debian) but problems in the internal libraries are preventing a successful cross-build - despite the (incomplete) summary on the freeswitch wiki page. Not sure how much more time to spend on this one.


• Various improvements in emdebian-tools - 1.4.1 is now in Lenny and these fixes will be uploaded to Emdebian. Depending on a few discussions here at DebConf, I am likely to make one more upload to Debian, possibly only as a debconf translation update.


• Continuing discussions about TDebs - more details to be sorted out during DebConf itself in one or more of the 4 debian i18n sessions.

Overall, very good.
(Posting delayed due to network problems.)

It appears that drivel cannot cope with posting to the blog from DebConf.

over 96% successful

243 packages built successfully.
8 packages failed to cross build.

Failing packages

1. cups
   

   http-addr.c: In function 'httpGetHostByName':
   
   http-addr.c:464: error: invalid 'asm': invalid operand for code 'w'
   
   http-addr.c:464: error: invalid 'asm': invalid operand for code 'w'

   
   history
   


2. gtk+2.0 - cairo-directfb problems.
   history
   The cross-build keeps finding /usr/arm-linux-gnu/lib/libcairo.so first so it misses the extra symbols provided in /usr/arm-linux-gnu/lib/libcairo-directfb/lib/libcairo.so. Hopefully, this can be resolved by judicious hackery of LDFLAGS and LD_LIBRARY_PATH.


3. ncurses
   
/trunk/n/ncurses/trunk/ncurses-5.6+20080713/ncurses/curses.priv.h:1417: warning: parameter names (without types) in function declaration
/trunk/n/ncurses/trunk/ncurses-5.6+20080713/ncurses/curses.priv.h:1418: error: expected '=', ',', ';', 'asm' or '__attribute__' before '_nc_to_widechar'

   history
   


4. ntp - same assembly error as cups
   

   authkeys.c: In function 'authencrypt':
   
   authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'
   
   authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'

   
   history
   


5. slang2 - confusing build behaviour - outside a chroot, files are put into a directory called elfobjs. Inside a chroot, files are put into a directory called elfamd64objs (and presumably that would change according to the BUILD architecture with is plain insane.
   history
   

The assembly errors look spooky and the cairo and slang2 errors could be made much simpler if the upstream builds were sane. (Honestly, why can't we just have /usr/lib/libcairo-directfb.so ?)

Some other packages fail to build, e.g. gcc-4.2, but we don't need those anymore so I'm ignoring those.

Autobuilder Summary:

138 packages built successfully. 19 packages failed to cross build. 2 packages are tagged for a manual upload. (Dependency issues - rather than break the archive, these packages will not be uploaded until relevant dependencies have been fixed and uploaded.)

Stats

87% built OK
12% failed

Notes:

1. All these packages built previously and fail at the latest versions in Debian Sid. Many could be upstream errors. All exist in Emdebian at
   previous versions (only 1 failure is known to contain bugs in the current versions in Emdebian and this only affects GUI installations. Two other GUI packages contains bugs but build OK.).
2. Some of the packages in the repository (including some of the failures) are not used in *any* of the default emsandbox configurations. Two need to be removed from the repository completely.
3. Some fixes are waiting for updates in Debian, either new uploads migrating to mirrors or pending fixes awaiting upload by me. (Probably tomorrow.)
4. The history of each package is now available via the Emdebian Autobuilder Report. The package name links to the history of previous build attempts for this package. The version string links to the repository data for the current version in Emdebian. The result of the build links to the build log.
5. A comparison with Debian unstable is constantly updated.
6. 254 source packages exist in Emdebian but only 137 are currently built by the autobuilder. The rest will be built when new versions are uploaded to Debian (in some cases, that won't be until after Lenny). If I get time, I'll add build logs for these packages.
7. The repository contains a fair number of packages that we don't currently use. This is a reflection of the problems of determining the "path" through the dependencies when starting at the bottom (glibc| uClibc etc.). Bear this in mind when considering new architectures and new package sets. The process involves a lot of trial and error, particularly as cross-building potentially alters the dependency path at each iteration.

Problems

1. libselinux and libsepol have just been uploaded with a bug fix that is needed for e2fsprogs and file cross-builds, amongst others. Waiting
   for the ARM mirror to receive the new builds, hopefully tomorrow.
2. gtk+2.0 fails to cross-build because the patches now try to build the udeb which comes up against a bug in dpkg-cross. I'll be uploading the new version (including a couple of other bug fixes) tonight. (Final testing currently in progress.). The Gtk package also needs some work to run /usr/lib/libgtk2.0-0/update-gdkpixbuf-loaders so that the icons can be read in the GUI.
3. pango1.0 also needs a fix to update the pango modules. This is usually done at build time but it means running a cross-built binary. It is a minor task and not CPU intensive so I plan to do this in postinst. Without this fix, no text is rendered in the GUI, only empty glyphs.
4. xfonts-base is way too large still. More work is needed here to optimise just what is meant by the 'fixed' font for X, precisely which font files can be removed, which might be needed outside C or English locales etc.
5. Other packages not currently cross-building include: (NOTE: Some of these build logs are VERY large.)
   • adduser - needs to be removed from the repository, it is perl and we implement it via busybox.
   • cups|cupsys - /usr/include/pthread.h:653: warning: '__regparm__' attribute directive ignored.
   • gcc-4.2 - needs to be removed from the repository, it no longer builds libgcc1 which was the main reason for cross-building it.
   • glib2.0 - more generated content: /bin/sh: line 2: /usr/bin/glib-genmarshal: No such file or directory
   • gnomevfs - hasn't built for ages and isn't usable anyway due to dependency issues. Currently preventing the use of gpe-filemanager. Due to be replaced by GIO in Debian.
   • libusb - build failure, so far resistant to usual fixes.
   • ncurses - build failure - uses gcc in wide char support build.
   • ntp - build failure - assembly error. authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'
   • pcre3 - build failure - apparently simply unresolved symbols resistant to usual fixes.
   
   
   • slang2 - possibly needs a patch update, tries to build -pic archives and then fails.
   • xorg-server - build failure: make[4]: *** No rule to make target `-L/usr/arm-linux-gnu/lib', needed by `Xprt'. Stop.

Of the problems above, only problems 2, 3 and 4 (gtk, pango and xfonts-base) are "critical" as these are the only ones where the current packages are sufficiently broken that the new version is essential before Emdebian can make a "release" of a series of three root filesystems for ARM. Sizes are still larger than desired - a fix for that requires changes in glibc which I hope to investigate during DebConf.

Any help on the above issues is appreciated.

Problem-solving:
$ emsource -c $package
$ cd /path/to/$package
$ emdebuild

If you have SVN access, use 'emdebuild --svn-only' to commit your changes (even if your changes don't solve all the problems).

Without SVN access, post the result of 'svn diff ../emdebian*' to this list (or 'svn diff ../debian*' if you have added a patch for the upstream code to debian/patches/).

Platforms:
All the above is based solely on Emdebian ARM for balloon3:
Linux balloon 2.6.25.2-pxa270 #1 Sun May 18 22:38:11 BST 2008 armv5tel unknown
balloon3 in CUED case
http://balloonboard.org/gallery/300/balloon3-0v1-fpga.jpg

AFAICT no other ARM platforms have been tested.

I have a few fixes for the balloon3-config package that also needs an update in Emdebian - I've had some feedback from the touchscreen driver
upstream that we might be trying to use the wrong device (/dev/input/mice is the wrong target for the symlink, we can set whatever device we want in /etc/X11/xorg.conf and the driver will use that in preference to /dev/event0). As this is device-specific, /etc/X11/xorg.conf will be put into balloon3-config which is available via Emdebian SVN.

Some time ago, I added a
Spammers Hall of Shame to my first website, codehelp.co.uk.

"... sites that send me unsolicited commercial email can expect a poison
response. If such companies are arrogant enough to not even read the
contents of this page, then I shall post details of their incompetence
here."

It's not often that I add a new one - generally when the spammer in question tries to claim that it is not spam and pretends to have taken an interest in the site when it is plainly obvious that the About page has been completely ignored. It takes a special kind of idiocy to get onto the list - something that motivates me to a noticeable level of annoyance. The vast majority of spam doesn't come anywhere close. It has to be something that has some indication of being supposedly "customised" for me and my website(s) yet equally takes absolutely no notice of the content of that website, specifically the content explicitly added to About that should have been noticed by anyone with half a brain.

Anyway, my latest poison pen is now available for everyone to see.

True, it gives their spam email a wider airing but it's not as if codehelp.co.uk is anything but a very quiet backwater of the internet so it's not going to affect traffic in any real way. I just get annoyed when idiots who claim to be professionals do not have the simple courtesy of reading.

In a similar tone:
http://qof-gen.sourceforge.net/#free

My first go at getting Emdebian to run a GUI:


Yes, yes, there are no icons and the touchscreen library doesn't work so
the apps can't start (at least via the screen) but it's a start!

Silver USB cable on the left if ethernet-over-usb (which works fine), on the right is a USB key from which the root filesystem, kernel and modules were installed (not needed for subsequent operation but used for debugging by copying log files to the key etc.).

Stats:
Still too large at over 80Mb installed for a full GUI but that is still
100Mb smaller than a minimal Debian system without any GUI stuff.
Available only for ARM
Tested only on Balloon3 : http://balloonboard.org/index.html
Not installing cleanly at this stage - some fettling is required.

192 packages used.
Linux balloon 2.6.25.2-pxa270 #1 Sun May 18 22:38:11 BST 2008 armv5tel
unknown

ii dpkg 1.14.20em1
ii libgpewidget1 0.115-5em1
ii libgtk2.0-0 2.12.3-2em1

Access:
Ethernet over USB (static IP only at the moment)
USB storage / keyboard
Serial connection
SSH pending

Lots to do to make it simple but at least it works.

Problems with hmrc.gov.uk - the new non-PDF format of the tax return is a complete mess and very misleading. I wanted a PDF so that I could compare the generated return with an actual return from previous years, so I sent this feedback:

"The Help and FAQ claim that a colour PDF is available and I'm
assuming that the PDF will be a closer match to the paper based form
which I need to be able to compare this return against last year. The
option to obtain a PDF has disappeared. (I am able to process PDF
without needing any software from Adobe as that is the main point of a
Portable Document Format so if this is because the site is checking for
Adobe software on my computer, it should stop doing so and simply
provide a link to the PDF file.) "

The response shows that nobody actually reads the comment:

"Thank you for your online enquiry.

To print your return in PDF format you are required to have Adobe Acrobat Reader 4.0 or above. If you do not have this on your computer please logon http://www.adobe.com/products/acrobat/readstep.html to download a copy. Please be aware that this download can take a long time."

i.e. Her Majesty's Government requires that you allow them to interrogate your computer to find a specific executable file or registry ID and if this is not found, certain services will be refused to you without further comment.

Instead of just generating the PDF on the server and providing a link to it.

The worst part is, the "secure gateway" used by HMRC doesn't even allow replies!

The only option is delete!!

More bias from government departments against free software. I refuse to use anything under an Adobe licence when there are perfectly capable PDF clients available under a free licence. (Let alone the discriminatory download licence). (I'm a British national but I still object to a licence that would prevent someone with dual British-Iraqi nationality, for example, obtaining a service from HMRC.) Besides which, I somehow doubt that the site would be able to correctly identify the Unix version of Adobe Acrobat Reader.

I receive notification of all my taxable income via PDF so why is the taxman different??