Codehelp blog

OK, I know I've been very quiet on the blog for weeks now but I have my reasons and all will become clear, eventually. I'm principally waiting for the Lenny release before being able to start full discussions about certain ideas that have been consuming a lot of my time recently. I continue to do what I can to get Lenny out of the door because the wait is driving me nuts. (Fixed another RC bug yesterday, yippee!)

Anyway, something I can talk about is Emdebian Grip. Emdebian has always wanted to support "in-between" devices that don't need the full-on cross-build everything approach of the current Emdebian packages for ARM but don't quite have room for a full Debian install either. Until I had at least a basic package set available via the cross-building system, I didn't want to start working on an intermediate system, for reasons of simplicity mainly, but also because I knew that getting a cross-building package set would involve changes to the tools needed for the intermediary set.

Bah, I'm waffling again. For the background, see Emdebian, Squeeze, Grip and Crush.

The good news is that Emdebian Grip, once running, will take far, far less time (for everyone) than the full-on cross-building malarkey of Emdebian Crush. Grip can be 99.9% automated by breaking Debian into discrete package sets or 'components'.

Componentising Debian for Emdebian

Emdebian Grip is about a smaller Debian, nothing more. No changes in functionality, no changes in dependencies; only changes in final installation size and Packages.gz size. It means using the code from dpkg-cross to unpack packages, remove unwanted files and repack the .deb. Yes, yes, this can be done one package at a time at a trivial level. Emdebian Grip seeks to bring a standard method for this operation that will work for all packages on any architecture for any architecture from any suite, creating and maintaining repositories of "gripped" packages for use in devices like my Acer Aspire1 where you just don't need documentation, manpages and the rest. Squeeze is the obvious target release but any release supported by the current version of dpkg could be "gripped" too, like oldstable.

The componentisation comes in at the level of the Packages.gz file. If the device cannot realistically be expected to run CAD software or has no fancy graphics or other power-hungry hardware, why should packages that need such resources clutter up the apt-cache list? This data isn't just slowing down searching for packages you do want, it slows down every installation process (that bit about 'Reading database . . . . . '), it slows down the process of getting the updated cache data in the first place because there is more of it, it wastes space on your device in several different places when the whole point of Emdebian is that disc space is not cheap and it wastes your precious battery power if you need to do simple package management stuff away from the mains. What is needed is a mini-repository, a mini-mirror that just contains the parts of Debian that your machine needs - a "well-trimmed" mirror with sources and all dependencies satisfied, a mirror that can be automatically updated but which doesn't keep getting NEW packages that you won't use. New dependencies will be added, yes. New packages unrelated to what you currently use? Absolutely not. You could even keep a mini-mirror on your home network, updating and "gripping" while you sleep, because a mini-mirror should be a lot, lot smaller than a standard Debian mirror.

I now have a script that will do the work, what I've lacked is the idea on how to "feed" packages to the Gripper.

So, I'm going to be exploring this method:

1. An arbitrary package set - say the output of 'dpkg -l' from the target machine setup by an expert, or 'dpkg --get-selections' or just the packages from a normal debootstrap - the basic idea is to start with only the packages you absolutely must have for your target device.


2. A very small wrapper for some debootstrap code that downloads those .debs from the Debian mirrors - debootstrap can cope with choosing the mirror, the architecture, the suite etc. (already written, landing in Emdebian SVN soon).


3. Parse the list of packages (.debs) and use 'apt-cache showsrc' to convert into a list of source packages. (Script done).


4. 'apt-get source' to match the source package to the binary. At this point, we have what amounts to a partial mirror but with very well defined limits. There are other ways of doing the same thing so anything that gets you to this point is as equal as the steps above, as long as the .debs exist alongside the .dsc and all associated source package files. What you need is full control over the list of packages at this stage, we'll worry about dependencies later.


5. Simple scripting parses the .dsc to create suitable debian/files content (done).


6. dpkg-genchanges regenerates the .changes file - critical for keeping the Gripped packages with the correct source packages in the final Grip repository


7. Now run 'emgrip' against the .changes file to convert the .debs, update the .changes file with the new, smaller, .deb details and copy everything into a suitable incoming/ directory.


8. point your repository software (tested with reprepro) at the incoming/ directory and generate your Gripped repository - which only contains the packages you listed at the start, just smaller. reprepro has a neat way of allowing foobar-1.2.3-2 to be included even if foobar-1.2.3-1 does not exist in the repository, as long as foobar_1.2.3.orig.tar.gz exists somewhere that reprepro can find and even though foobar-1.2.3-2.dsc does not mention the .orig.tar.gz. Neat! (Tested and working.)


9. Run 'edos-debcheck' against the repository and "catch-up" with whatever packages are needed to make the repository as a whole "trimmed" (no unresolved dependencies).


10. Keep the update script running as a cron task to pull updated packages from incoming.debian.org, run the .changes through emgrip and update your repository.


11. Clean out the .debs, leave the source packages and restart the process for another architecture. (Crafty!)


12. Possibly, to support those who can afford to do a small amount of compilation on their Emdebian Grip device, add a second repository that provides the Build-Depends that may be missing from the initial package set - stuff like automake and pkgconfig. Depending on how that works out, it might not be many packages and it may be easier to fold them into the mini-mirror anyway.


13. Instead of 20,000 packages, I'm expecting something closer to 2,000, per mini-mirror.


14. The process of "gripping" a package also drops numerous control fields (like Homepage and VCS*), as well as Recommends, Suggests and Enhances as well as truncating the long description at a maximum of four lines - all in the name of reduced package data sizes.


15. The end result is that each mini-mirror contains a component package set for Emdebian Grip, using Debian packages. Mini-mirrors are still "well-trimmed" (no unresolved dependencies) because you get to choose which packages you include and if a particular package tries to bring in both Gtk and KDE dependencies, you can choose to omit that package and use a different one.


16. Whether Emdebian Grip will or should support dropping certain binary packages from the repository is a difficult issue - Emdebian Crush is probably the preferred route if you need to start optimising an Emdebian Grip mini-mirror that harshly.


17. Another wrapper will be needed to clean out cruft (old dependencies, outdated packages etc.) - that one may be harder to achieve by automation.

The first Emdebian Grip repositories will be for ARM, armel and i386, based on an XFCE environment package set.

"Gripping" all the packages in my /var/cache/apt/archives (about 1Gb) took about 30 minutes on a fast laptop that was doing other things at the time - the real packages will be "gripped" on the Emdebian server and for multiple architectures in sequence. It's certainly conceivable that any number of mini-mirrors can exist with any number of package sets (components), all with "gripped" packages. You can choose to "grip" .debs upon download from a normal Debian mirror but that wastes time if there is more than one device, it wastes space in /var/lib/apt/lists and /var/lib/dpkg/ and wastes bandwidth. For portable devices that you want to be using, not updating, I think that a mini-mirror is a worthwhile bonus.

Expect a 20% reduction in installation size (for more, you'll need the cross-building, functionality-modifying Emdebian Crush shebang) but a large decrease in the size of all the apt-cache data and dpkg status data. I'm hoping to push towards 30% reductions but most of these figures are guesswork right now.

And beyond the mini-mirrors? Debian-Installer, of course. I really need to set some time aside to prepare an Emdebian Grip installer using current D-I. Simple things like:

• Preferring ext2 instead of ext3 for solid state file systems,


• preferring XFCE to GNOME for the Desktop System,


• turning off Recommends before even the first package gets chosen - let alone installed,
  
• supporting machine:variant customisation scripting so that particular devices can be configured from first boot instead of having to rely on defaults or manual fixes via Wiki pages.

It really, really annoys me that I have to spend ages getting devices like the Acer Aspire1 working properly under Debian and that I have to take particular care during the installer to change the Debian default to something more sane for that kind of device. For Debian to be The Universal Operating System, the installer needs to be as smooth on an NSLU2* as on a standard desktop, a fast server and a cluster. This isn't just an architecture thing, there are big and small devices in all architectures, it needs to be a machine:variant thing. Mini-mirrors of single components are a start and part of the bigger answer is to support machine-specific configuration packages in these mini-mirrors. Packages that only exist in a specific mini-mirror and are specifically designed to setup a host of machine-specific settings, blacklist certain modules and enable others, set sensible settings in /etc/fstab and other places based on results from things like powertop, turning off features for some devices and turning them on for others. The whole point of such packages is that what they do would be insane on what would currently be considered a "typical Debian machine".

*(I'm not talking about an NSLU2 with an external hard drive here, I'm talking about fitting Emdebian Crush into the 6.4Mb of space on the NSLU2 itself so that external hard drives can be hot-plugged, as well as devices like the OpenMoko and iPAQs etc.)

Don't get me wrong, D-I does a fantastic job for the kinds of devices Debian currently supports but there are a host of other devices out there that need customised solutions where a mini-mirror and a machine-specific configuration package can remove huge amounts of hassle from the actual installation process. Emdebian Crush already uses machine:variant specific packages and the current Emdebian packages repository is, itself, a mini-mirror, albeit not well trimmed.

Let's make the most of "Debian Squeeze" - let's squeeze it onto more devices and make it even easier to get Debian onto anything.

It's really quite strange to come across a tool that is incredibly useful, does exactly what I need but of which I was completely unaware. This time it is manpages.debian.net - how long has this been around and why hasn't it been announced somewhere?
The best part is that if a manpage uses a See also section, the references to other manpages get turned into HTML links to the relevant manpage on the site.
Getting a manpage is now as simple as:

http://manpages.debian.net/cgi-bin/man.cgi?query=%s

which makes it trivial to put this onto the epiphany smart bookmark toolbar.
The site supports:

• keyword search via apropos,
• showing manpages from each distribution, default is Sid
• rendering manpages in other formats like PDF
• translated manpages, if the translation exists.
• manpages via section

Thanks to Patrick Schoenfeld for inadvertently leading me to the site - I couldn't find who created the site so step up and be known.

Update

To those making comments on this entry, please note:
I have nothing directly to do with the site, I just found it and blogged about it so don't use my blog as a bug list. Ta. Such comments have been deleted and further comments are now disabled for this entry.

It's been really strange coming back from DebConf then Extremadura and having to fit my code time around paid work and housework and general stuff again. I keep wondering why I'm not getting things done and then realising that I've only got so many hours a week nowadays compared to the entire week during the worksessions. What makes it worse is the short gap between the conferences. I misjudged the length of time without income and didn't make any days available for paid work between DebConf and Extremadura. OK, it was only about a week but it was a further week without income followed by the week getting to and from Spain.
I'm now doing extra days to recover from the extra time without income and it makes the contrast even more noticeable. The tiredness from the extra work then makes it hard to get started on what I thought I'd get done. I've done a little bit on emdebian-tools and apt-cross, I've done a little bit on a few RC bugs and I've not done a load of stuff I thought I would have done by now.
If you're wondering why something I said I'd do has not been done, file a bug and I'll get to it. (The bug lists for buildd.emdebian.org or emdebian-tools are my main ToDo lists, if it appears in that list of bugs, I'll not forget. If it appears in an email, I probably will forget.)

Linpus Lite (or maybe just aspire one mail) is only going to last a few more days on this aspire1, it's beginning to annoy me:

Update

sylpheed has now replaced aspire mail. OK, I still use aspire mail for calendar and addressbook stuff (for now), but sylpheed is a much better email client.

1. aspire one mail has no GnuPG support


2. aspire one mail has no non-HTML composition support. No, really, it doesn't even send plain text alongside the HTML. email sent from the aspire to myself comes in as an attachment in other email clients! (i.e. spam)

So, I would have to investigate a different email client but then there are niggles with Linpus itself:

1. the aspire default kernel build lacks the pl2303 kernel module so that I can't use the aspire to control my balloon3 board over usb serial


2. intermittent problem with network manager forgetting WPA passphrases

Testing continues - probably resulting in installing Lenny at the end of this week. Some of these things might be fixable using pages like taoofmac and Reg Hardware.
Unfortunately, the problem with usbserial means that I'll have to take two laptops if I want to demonstrate the balloon3 board in places.
I also don't seem to be able to locate any copyright files which makes me nervous. Maybe it's just a lack of knowledge of Fedora (at least not since FC5).

OK, it's still running Linpus Lite and yum but that's enough to get things like drivel and ssh installed. Add in xchat-gnome and it's quite usable. The whole machine bounces around quite insanely when I'm typing this blog entry on my lap, though.
pilot-qof and a bit of XSL from the data-freedom package from the same source, allows easy import of ICS calendar data and VCF contact data.
Aspire Mail is clearly based on evolution.
I'll live with Linpus (sounds freakishly similar to lupus - or maybe that is just my medical bias) for a month or so - I'll get Debian on this sometime before my return to Extremadura.

A few speakers at DebConf8 used gobby during their talks to collate ideas without breaking the flow of the session.
I've just come back from the Emdebian meeting in Extremadura (of which more later) where we also used Gobby - FAI used it too. Once the network setup was working, it was simply a case of clicking Create Session in gobby on my laptop and everyone else clicking Join Session, entering the IP of my laptop (only accessible locally) and choosing a colour.
Gobby is real-time, currently plain-text only but clearly indicates how sections have been edited and has an integrated IRC box for each session. At the end of the meeting, every gobby user has a copy of every document they opened in every session they joined. updated to the moment they quit the session.
Based on using gobby in events like this, I'd strongly recommend that someone at each worksession-type meeting starts a gobby session and maintains that session for the duration of the meeting.
Gobby even does syntax highlighting - depending on how many people join and the colours they choose, this might become difficult so it can be turned off too.
There are only a few minor issues:

1. The colour bar warning box isn't particularly intuitive - you need to click on what looks like a dimmed progress bar to select a unique colour for yourself.


2. I initially thought that Gobby had no authentication - in a local meeting, anyone who can connect to the specified IP address is able to edit the document. Since then, I've noticed it does support password authentication and there is mention of fully encrypted connections so it may be possible to use it over public networks.


3. According to Wikipedia, there is a gobby server that allows non-GNU/Linux clients to run gobby and join any gobby session - I didn't explore this and the Debian package doesn't mention anything about a server option or mixed-OS support.
   
   • Update - the server is called Sobby and is in Debian.
   
   


4. The right hand side of the window is a bit wasted - maybe some kind of sidebar would be handy, especially docking the userlist into the window so that edits can be identified more quickly.

The current method of patching into place a series of ./configure variables as $(DEB_HOST_GNU_TYPE).cache files is becoming a pain - it blocks native builds of Emdebian packages for a start. So, I've been experimenting with a different method of support.

Centralised cache value support

Initially, I've collated all the current cache values needed (i.e. beyond the ones already provided for specific architectures) into /etc/dpkg-cross/cross-config.cache and have a script in emdebian-tools called emcache that will manage the central values by comparing them to the real values used in each build tree.
So, cache values, what are they?

1. Cross-builds fail if ./configure tries to compile a test program and then try to execute it because it will have been compiled with the cross-compiler - instead, the cross-build passes the value that the test was trying to calculate so that ./configure does not need to try to execute any cross-compiled code.


2. The values are derived from native builds, written to a $(DEB_HOST_GNU_TYPE).cache file that is passed to ./configure via debian/rules.


3. The values turn out to be mostly (completely?) architecture-independent, so are eminently suitable for collation into a central cache file.


4. Quite package-specific. There is some overlap but most are customised for a specific package. This led to a development of the central cache file: package-maintained cache files to support extensibility.

Package maintained cache files

I'm now experimenting with cache files written (and maintained) by packages during normal builds and dropped into /etc/dpkg-cross/cross-config.d/ - which would later migrate into dpkg support. Ideas welcome!
The basic idea will be that IF cross-building, the build process would check for /etc/dpkg[-cross]/cross-config.d/$source_package and, if found, add that to the cache value list for that build.
Cache files are shell scripts so this it be manageable to bring some intelligence to the process.

Where does MIPS come in?

Well, all the builds, so far, have been ARM or armel (where the cache values are likely to be identical) and i386 (which doesn't need a cross-compiler at all and therefore does not need cache files). MIPS is the first chance to check the cache values and ensure that the above criteria really are true.
Now that we have a volunteer looking at MIPS, the current MIPS builds are a suitable test bed.

What does this have to do with Debian?

Once Lenny is released, I'll be looking to implement this support via the long term mass bug filing for cross-build support into the packages themselves. It is possible that the process can be entirely automated - as long as the script knows which values are relevant. A normal cache file, post-build, can be very, very long whereas most cache files for Emdebian are just a few lines long - the longest is about 20 lines.
The current process is outlined on the Emdebian website, including a list of packages that currently need cache files to cross-build.
Oh, and just a note - when I start this phase of the long term mass bug filing, I'll first do delayed NMUs for the existing bug reports - hence the delay until Lenny is released.
In order to speed things along, I have also been doing some RC bugs - downgrading some, fixing others etc. (I've finished 6 since coming back from DebConf, so far, IIRC.) All efforts to get Lenny released will help Emdebian too, so if you want to help Emdebian, then help Lenny get out of the door and let's get to work on Squeeze.

(Squeeze is a great name for a release with lots of fixes for embedded stuff, don't you think?)

I've written a simple perl script using SOAP to add some useful data to the RC bug list (or any list of bugs). It can easily be extended but right now it supports simple things like the date the bug was filed, short description of the package, the name of the maintainer, the most recent upload data, the debtags for the package, apt-cache policy data and whether the bug is already tagged pending.
Zenity presents this info in a question dialog, clicking OK loads the bug and the PTS in your browser using 'sensible-browser'.
Clicking cancel offers a new zenity dialog asking for the next bug number. (Click Cancel there to end the script.)
Code: http://linux.codehelp.co.uk/rc-zenity.pl (GPLv3 or later)
POD: http://linux.codehelp.co.uk/rc-zenity.html (easily convertible into a manpage using pod2man)
If you like it, use it.
If lots of people like it, maybe it can go into an existing package.
If I've duplicated another script that I just didn't know about, tell me.

It's not exclusively for RC bugs, it could work with any bug list or be modified to only look for particular severity settings.
Thanks to Don Armstrong for making debbugs so accessible using SOAP. The debbugs SOAP implementation is just so straightforward with a whole load of useful data returned without needing repeated queries, it makes scripts like this really satisfying.