Codehelp blog

OK, it's gone 2am and I'm still battling with what was meant to be a simple task - check the latest (unreleased) changes in tslib on my touchscreen device.

First problem was that I forgot that the balloon is running Lenny, so I suddenly needed the armel libc6 from Sid. OK, that's not bad, a bit of a pain to have to install it but less pain than creating a Lenny chroot to cross-build tslib - marginally.

Second problem was more awkward - turns out that emdebian-tools 1.6.1 has broken the more fragile cross-build support which carefully wraps broken libtool installations. I was intending to work out how to fix each one of those before Squeeze anyway. The fix for the workaround was relatively straightforward and will be in 1.6.2, due for release once I've had some sleep.

Third problem was something much older - turns out that this broken libtool wrapper had only been able to work for the default cross-building architecture since about emdebian-tools 1.4 or 1.5 and if asked to build for a different arch, it got the wrong symbols. Hmm.

The diffs for each of these changes appear simple but, as ever, identifying the cause of the bug was harder. (In this case, sometimes the error showed up as "C++ compiler unable to create executables" - thanks autotools, that's really handy to know; other times it showed up as ERROR, symbols from ./.libs/libfoo are ABI 4 but symbols from libc are ABI version 0 - still cryptic but at least it is entirely accurate.)

Still, I've now been able to test tslib 1.0-7 and it works just fine and I've got an extra upload of emdebian-tools 1.6.2 to make tomorrow. Ah well.

This is my first entry from my Acer Aspire1 since installing the latest Emdebian Grip and the 2.6.28 kernel that now provides free wireless support. A few tweaks still needed to get a few packages sorted out so I've got a Debian source on this box too. Still, with my usual packages (iceweasel, sylpheed, grisbi, geany), all my email cached data, liferea data, a few documents and a few Debian packages and I'm still at only 753Mb used.

Using wicd from squeeze, I've got wireless working - there's no led lit but I'm sure that's fixable.

Copying a few files around from ~/.gnome2/ on my other laptop and suddenly I have drivel, sylpheed and liferea all pre-configured.

I'll be blitzing this install again numerous times in the next few days but by keeping the ~/.gnome2/ files on a USB stick, it will be simple to restore the system, especially as I'm using IMAP email.

(If you're wondering, the drivel upload is delayed by the churn in Debian Sid - waiting for the libmtp7|8 transition to make rhythmbox available in a clean chroot.)

I'm pleased to make Emdebian Grip unstable available - a smaller, binary-compatible, Debian without manpages, without unwanted translation files (uses Emdebian TDebs) and without package documentation. The current packages will be migrated into Emdebian Grip testing in due course and then be released alongside Lenny.
Emdebian now has two flavours with Grip taking the intermediate position in terms of size whilst retaining full compatibility with Debian. Grip "cherry-picks" packages from Debian but retains the unchanged Debian sources - the main binary and library packages will exist but -dev, -doc and -dbg packages may well be missing, as well as some of the more unusual OR dependencies).

I've just converted my Acer Aspire1 (i386) - running XFCE - from Debian Lenny to Emdebian Grip:
607 upgraded, 1 newly installed, 2 to remove and 0 not upgraded.
Need to get 184MB of archives.
After this operation, 278MB disk space will be freed.

Yes, update 600 packages and have nearly 300Mb more free space - that's a 400Kb space saving per package, on average. Gaining 90Mb more free space than the total size of the download is nice too. Comparisons are after using 'sudo apt-get clean' to remove downloaded archives:
Debian Lenny:
/dev/sda1 7.1G 1.2G 5.6G 17% /
Emdebian Grip (unstable)
/dev/sda1 7.1G 919M 5.8G 14% /

I make that about 25% smaller and not all packages installed on the machine have been "gripped" - (about 100 remain, many of which I should probably remove anyway).
Method: See http://www.emdebian.org/grip/index.html.

Add the Grip source to /etc/apt/sources.list.d/
deb http://buildd.emdebian.org/grip/ unstable main
$ sudo apt-get update
$ sudo apt-get install grip-config

This is a very important stage - Emdebian Grip is still in development and with the Lenny freeze, a few wrappers and helpers are needed to allow smooth migrations between Debian and Emdebian - grip-config is Architecture:all and contains the relevant scripts. (grip-config is Priority: required so debootstrap picks it up by default - this also means that grip-config cannot be part of Debian.) grip-config also includes the same key as the emdebian-archive-keyring package used by all Emdebian repositories.

Only after grip-config is installed should you use:
$ sudo apt-get dist-upgrade

Note that this is a complete replacement, including libc6, apt, coreutils, dpkg and (on i386) the stock kernel. As such, it can take a while to complete the installation once the download is complete. (As ever, if the kernel is updated, ensure you reboot before you next suspend to disk - even the minimal "gripping" changes to the kernel package may confuse grub when deciding whether to use the suspended image.)

A few packages are currently behind Debian Sid (notably ncurses) and I'm working on the scripts to enable full update automation.

One other problem is that /usr/share/locale/locale.alias has also disappeared which compromises localisation support right now but that's fairly easy to fix. (langupdate is also missing so there is no effective localisation support right now - I need to cross-build langupdate for the grip architectures.)
Note that this test install adds the Emdebian Grip repository to the existing Debian sources and relies on the em[0-9] version suffix to implement the upgrade - packages that are behind Debian get left at the Debian versions, you can also use normal pinning.

$ apt-cache policy
Package files:
100 /var/lib/dpkg/status
release a=now
500 http://buildd.emdebian.org unstable/main Packages
release v=0.1,o=Debian,a=sid,l=EmdebianGrip,c=main
origin buildd.emdebian.org
500 http://ftp.uk.debian.org sid/main Packages
release o=Debian,a=unstable,l=Debian,c=main
origin ftp.uk.debian.org
Pinned packages:

I've done some testing with debootstrap and a chroot works too - it should be possible to ally these packages with the normal Debian installer because there is no point "gripping" udebs that are already v.small and the packages that actually get installed are dictated by your choice of mirror. debian-live should be gaining support for Emdebian Grip in due course.

If someone fancies trying a test installation by choosing http://buildd.emdebian.org/grip as their mirror in d-i, let me know.

(Installing Grip is likely to result in further size gains as the Grip repository data does not support Recommends, so you don't have to try and get d-i to not use recommended packages).

Emdebian Grip (unstable) is now available for 7 architectures:

i386, amd64, arm, armel, mips, mipsel and powerpc.
(amd64 is only supported for debugging purposes, most of my development machines are amd64.)
Please report any and all bugs to the buildd.emdebian.org pseudo-package in the Debian BTS.

If you want more packages added, set up a Debian machine or chroot with all the packages you need (and only the ones you need) and send the
complete output of 'dpkg --get-selections' to the debian-embedded mailing list. Alternatively, file RFP bugs against buildd.emdebian.org at wishlist severity.

To see the current list of packages, use the search support

Headline packages:

coreutils, perl, python, apt, dpkg, grip-config, XFCE, debhelper, make, devscripts, sylpheed, iceweasel, drivel, xchat-gnome, gpe-tetris,
totem-gstreamer, liferea, thunar, evince, seahorse, gthumb, libsqlite3, pilot-link, python-gnome2, lua, tcl, gcc-4.3, binutils, libmysqlclient15off.

Currently, very few -dev packages exist in Grip. I'll make the emdebian-grip package available on Grip itself in due course so that new and bespoke packages can also be "gripped".

I'm migrating the current packages to Emdebian Grip testing soon and I'll be making a release of Emdebian Grip 1.0 (based on Debian 5.0 "Lenny") as well as Emdebian Crush 1.0 (based on Debian 5.0 "Lenny"), alongside the main Lenny release.

Finally, note that Grip is binary compatible with Debian - indeed, the binaries themselves are untouched - but bugs that appear in Emdebian Grip must only be filed against the buildd.emdebian.org pseudo-package. Right now, reportbug is not part of Emdebian Grip because I need a way of telling reportbug to only use the pseudo-package.

The main usability improvements will come in the development of Emdebian 2.0 (based on Debian 6.0 "Squeeze") - please consider Emdebian 1.0 as a "developer release", think back to what things were like with Slackware 1.0 or Debian buzz, rex or bo.

P.S. there is no concept of a release-critical bug in Emdebian Grip (at least, not yet) - the 1.0 release will happen as soon as Grip testing is in sync with the released Lenny (no matter what). Please do not file any bugs against buildd.emdebian.org with higher severity than important, it will just annoy people trying to fix RC bugs in Lenny.

If you are a maintainer in Debian and you inadvertently receive a bug report listed as 'found' in a version that ends in em[0-9], please reassign to buildd.emdebian.org, at severity no higher than important. Thanks.

The current method of patching into place a series of ./configure variables as $(DEB_HOST_GNU_TYPE).cache files is becoming a pain - it blocks native builds of Emdebian packages for a start. So, I've been experimenting with a different method of support.

Centralised cache value support

Initially, I've collated all the current cache values needed (i.e. beyond the ones already provided for specific architectures) into /etc/dpkg-cross/cross-config.cache and have a script in emdebian-tools called emcache that will manage the central values by comparing them to the real values used in each build tree.
So, cache values, what are they?

1. Cross-builds fail if ./configure tries to compile a test program and then try to execute it because it will have been compiled with the cross-compiler - instead, the cross-build passes the value that the test was trying to calculate so that ./configure does not need to try to execute any cross-compiled code.


2. The values are derived from native builds, written to a $(DEB_HOST_GNU_TYPE).cache file that is passed to ./configure via debian/rules.


3. The values turn out to be mostly (completely?) architecture-independent, so are eminently suitable for collation into a central cache file.


4. Quite package-specific. There is some overlap but most are customised for a specific package. This led to a development of the central cache file: package-maintained cache files to support extensibility.

Package maintained cache files

I'm now experimenting with cache files written (and maintained) by packages during normal builds and dropped into /etc/dpkg-cross/cross-config.d/ - which would later migrate into dpkg support. Ideas welcome!
The basic idea will be that IF cross-building, the build process would check for /etc/dpkg[-cross]/cross-config.d/$source_package and, if found, add that to the cache value list for that build.
Cache files are shell scripts so this it be manageable to bring some intelligence to the process.

Where does MIPS come in?

Well, all the builds, so far, have been ARM or armel (where the cache values are likely to be identical) and i386 (which doesn't need a cross-compiler at all and therefore does not need cache files). MIPS is the first chance to check the cache values and ensure that the above criteria really are true.
Now that we have a volunteer looking at MIPS, the current MIPS builds are a suitable test bed.

What does this have to do with Debian?

Once Lenny is released, I'll be looking to implement this support via the long term mass bug filing for cross-build support into the packages themselves. It is possible that the process can be entirely automated - as long as the script knows which values are relevant. A normal cache file, post-build, can be very, very long whereas most cache files for Emdebian are just a few lines long - the longest is about 20 lines.
The current process is outlined on the Emdebian website, including a list of packages that currently need cache files to cross-build.
Oh, and just a note - when I start this phase of the long term mass bug filing, I'll first do delayed NMUs for the existing bug reports - hence the delay until Lenny is released.
In order to speed things along, I have also been doing some RC bugs - downgrading some, fixing others etc. (I've finished 6 since coming back from DebConf, so far, IIRC.) All efforts to get Lenny released will help Emdebian too, so if you want to help Emdebian, then help Lenny get out of the door and let's get to work on Squeeze.

(Squeeze is a great name for a release with lots of fixes for embedded stuff, don't you think?)

over 96% successful

243 packages built successfully.
8 packages failed to cross build.

Failing packages

1. cups
   

   http-addr.c: In function 'httpGetHostByName':
   
   http-addr.c:464: error: invalid 'asm': invalid operand for code 'w'
   
   http-addr.c:464: error: invalid 'asm': invalid operand for code 'w'

   
   history
   


2. gtk+2.0 - cairo-directfb problems.
   history
   The cross-build keeps finding /usr/arm-linux-gnu/lib/libcairo.so first so it misses the extra symbols provided in /usr/arm-linux-gnu/lib/libcairo-directfb/lib/libcairo.so. Hopefully, this can be resolved by judicious hackery of LDFLAGS and LD_LIBRARY_PATH.


3. ncurses
   
/trunk/n/ncurses/trunk/ncurses-5.6+20080713/ncurses/curses.priv.h:1417: warning: parameter names (without types) in function declaration
/trunk/n/ncurses/trunk/ncurses-5.6+20080713/ncurses/curses.priv.h:1418: error: expected '=', ',', ';', 'asm' or '__attribute__' before '_nc_to_widechar'

   history
   


4. ntp - same assembly error as cups
   

   authkeys.c: In function 'authencrypt':
   
   authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'
   
   authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'

   
   history
   


5. slang2 - confusing build behaviour - outside a chroot, files are put into a directory called elfobjs. Inside a chroot, files are put into a directory called elfamd64objs (and presumably that would change according to the BUILD architecture with is plain insane.
   history
   

The assembly errors look spooky and the cairo and slang2 errors could be made much simpler if the upstream builds were sane. (Honestly, why can't we just have /usr/lib/libcairo-directfb.so ?)

Some other packages fail to build, e.g. gcc-4.2, but we don't need those anymore so I'm ignoring those.

Autobuilder Summary:

138 packages built successfully. 19 packages failed to cross build. 2 packages are tagged for a manual upload. (Dependency issues - rather than break the archive, these packages will not be uploaded until relevant dependencies have been fixed and uploaded.)

Stats

87% built OK
12% failed

Notes:

1. All these packages built previously and fail at the latest versions in Debian Sid. Many could be upstream errors. All exist in Emdebian at
   previous versions (only 1 failure is known to contain bugs in the current versions in Emdebian and this only affects GUI installations. Two other GUI packages contains bugs but build OK.).
2. Some of the packages in the repository (including some of the failures) are not used in *any* of the default emsandbox configurations. Two need to be removed from the repository completely.
3. Some fixes are waiting for updates in Debian, either new uploads migrating to mirrors or pending fixes awaiting upload by me. (Probably tomorrow.)
4. The history of each package is now available via the Emdebian Autobuilder Report. The package name links to the history of previous build attempts for this package. The version string links to the repository data for the current version in Emdebian. The result of the build links to the build log.
5. A comparison with Debian unstable is constantly updated.
6. 254 source packages exist in Emdebian but only 137 are currently built by the autobuilder. The rest will be built when new versions are uploaded to Debian (in some cases, that won't be until after Lenny). If I get time, I'll add build logs for these packages.
7. The repository contains a fair number of packages that we don't currently use. This is a reflection of the problems of determining the "path" through the dependencies when starting at the bottom (glibc| uClibc etc.). Bear this in mind when considering new architectures and new package sets. The process involves a lot of trial and error, particularly as cross-building potentially alters the dependency path at each iteration.

Problems

1. libselinux and libsepol have just been uploaded with a bug fix that is needed for e2fsprogs and file cross-builds, amongst others. Waiting
   for the ARM mirror to receive the new builds, hopefully tomorrow.
2. gtk+2.0 fails to cross-build because the patches now try to build the udeb which comes up against a bug in dpkg-cross. I'll be uploading the new version (including a couple of other bug fixes) tonight. (Final testing currently in progress.). The Gtk package also needs some work to run /usr/lib/libgtk2.0-0/update-gdkpixbuf-loaders so that the icons can be read in the GUI.
3. pango1.0 also needs a fix to update the pango modules. This is usually done at build time but it means running a cross-built binary. It is a minor task and not CPU intensive so I plan to do this in postinst. Without this fix, no text is rendered in the GUI, only empty glyphs.
4. xfonts-base is way too large still. More work is needed here to optimise just what is meant by the 'fixed' font for X, precisely which font files can be removed, which might be needed outside C or English locales etc.
5. Other packages not currently cross-building include: (NOTE: Some of these build logs are VERY large.)
   • adduser - needs to be removed from the repository, it is perl and we implement it via busybox.
   • cups|cupsys - /usr/include/pthread.h:653: warning: '__regparm__' attribute directive ignored.
   • gcc-4.2 - needs to be removed from the repository, it no longer builds libgcc1 which was the main reason for cross-building it.
   • glib2.0 - more generated content: /bin/sh: line 2: /usr/bin/glib-genmarshal: No such file or directory
   • gnomevfs - hasn't built for ages and isn't usable anyway due to dependency issues. Currently preventing the use of gpe-filemanager. Due to be replaced by GIO in Debian.
   • libusb - build failure, so far resistant to usual fixes.
   • ncurses - build failure - uses gcc in wide char support build.
   • ntp - build failure - assembly error. authkeys.c:445: error: invalid 'asm': invalid operand for code 'w'
   • pcre3 - build failure - apparently simply unresolved symbols resistant to usual fixes.
   
   
   • slang2 - possibly needs a patch update, tries to build -pic archives and then fails.
   • xorg-server - build failure: make[4]: *** No rule to make target `-L/usr/arm-linux-gnu/lib', needed by `Xprt'. Stop.

Of the problems above, only problems 2, 3 and 4 (gtk, pango and xfonts-base) are "critical" as these are the only ones where the current packages are sufficiently broken that the new version is essential before Emdebian can make a "release" of a series of three root filesystems for ARM. Sizes are still larger than desired - a fix for that requires changes in glibc which I hope to investigate during DebConf.

Any help on the above issues is appreciated.

Problem-solving:
$ emsource -c $package
$ cd /path/to/$package
$ emdebuild

If you have SVN access, use 'emdebuild --svn-only' to commit your changes (even if your changes don't solve all the problems).

Without SVN access, post the result of 'svn diff ../emdebian*' to this list (or 'svn diff ../debian*' if you have added a patch for the upstream code to debian/patches/).

Platforms:
All the above is based solely on Emdebian ARM for balloon3:
Linux balloon 2.6.25.2-pxa270 #1 Sun May 18 22:38:11 BST 2008 armv5tel unknown
balloon3 in CUED case
http://balloonboard.org/gallery/300/balloon3-0v1-fpga.jpg

AFAICT no other ARM platforms have been tested.

I have a few fixes for the balloon3-config package that also needs an update in Emdebian - I've had some feedback from the touchscreen driver
upstream that we might be trying to use the wrong device (/dev/input/mice is the wrong target for the symlink, we can set whatever device we want in /etc/X11/xorg.conf and the driver will use that in preference to /dev/event0). As this is device-specific, /etc/X11/xorg.conf will be put into balloon3-config which is available via Emdebian SVN.

My first go at getting Emdebian to run a GUI:


Yes, yes, there are no icons and the touchscreen library doesn't work so
the apps can't start (at least via the screen) but it's a start!

Silver USB cable on the left if ethernet-over-usb (which works fine), on the right is a USB key from which the root filesystem, kernel and modules were installed (not needed for subsequent operation but used for debugging by copying log files to the key etc.).

Stats:
Still too large at over 80Mb installed for a full GUI but that is still
100Mb smaller than a minimal Debian system without any GUI stuff.
Available only for ARM
Tested only on Balloon3 : http://balloonboard.org/index.html
Not installing cleanly at this stage - some fettling is required.

192 packages used.
Linux balloon 2.6.25.2-pxa270 #1 Sun May 18 22:38:11 BST 2008 armv5tel
unknown

ii dpkg 1.14.20em1
ii libgpewidget1 0.115-5em1
ii libgtk2.0-0 2.12.3-2em1

Access:
Ethernet over USB (static IP only at the moment)
USB storage / keyboard
Serial connection
SSH pending

Lots to do to make it simple but at least it works.

http://www.emdebian.org/buildd/

The scripts are in emdebian-tools 1.1.6 and with a few more fixes and tweaks, that will be the main body of v1.2.0 for Debian unstable.

A few issues with ssh and sudo passphrase caching vs running buildd scripts as root - I'll need to improve the documentation of the process before any other buildds can be set up. The need to have emdebian-tools and subversion installed inside the buildd chroot does make things a little more complicated than a standard wanna-build implementation - as does the need to download, build, install and clean-up cross dependencies.

This autobuilder is the Emdebian target package buildd - Hector Oron has another to build the cross-building toolchains. The toolchain autobuilder has fewer packages but each package is larger and is built for more architectures. Adding more architectures to the target autobuilder is quite difficult - packages that use cache files will need new cache files with different values for certain cache values. See the wiki for info on how to fix cache files. Each solves a different problem within the overall design of any buildd. In each case, instead of listening to email from incoming.debian.org, each buildd picks up the incoming data from the apt cache - a useful little time delay that gives me time to head-off certain problematic updates.

The target package autobuilder runs incrementally - a package is repeatedly rebuilt until it works and is then left alone until the next Debian release. There's lots of work available on the frontend if anyone fancies helping with the PHP. (See Emdebian SVN).

The advantage now is that as more packages fix their cross building bugs, more packages will autobuild without intervention which frees up more of my time to fix the more complicated problems. Equally, I can refer to the buildd logs in bug reports, helping everyone see what is going on.