Codehelp blog

Let's start with a caution: this is a work-in-progress and is aimed at Squeeze (for the tools) and Squeeze+1 for the actual TDebs. Nothing about TDebs will happen within Debian any time soon. DebConf9 could be an important step in the work - as long as we've released Lenny by then.
Various steps and decisions have been discussed and tested during the meeting, to the point where we now have an early draft specification for tdebs.
See also details of the ftpmaster meeting and progress on i18n and tdebs.
The draft spec includes links to the current resources, including links to git repositories for the changes being tested for dpkg and debhelper.
These changes will supersede the existing experimental scripts in the emdebian-tdeb package.
The spec will continue to be updated during the rest of this meeting and I'll continue working on the dpkg and debhelper support.
(A trivial patch for CDBS support is already done, dependent on the debhelper support.)
Next on my ToDo list is Emdebian Grip - including a few tweaks to em_installtdeb.

Following from unexpected results, I can't make up my mind why Gtk+ supports rotated labels.
Try it, load Glade3, create any normal Gtk window and add or select a standard GtkLabel, add some text. In the properties of the label, there is a spin button for Angle:. Why is that? Which applications have need for this? I can't think of any examples that I've seen in Debian. Does anybody use rotated labels?
I could possibly understand rotations in increments of 90 degrees but this supports rotation at an precision of two decimal places! I mean, why support a rotation of 89.27 degrees?
Even if this was a version of a progress bar, why is it supported for every GtkLabel anywhere?
Weird.
P.S. Off to Extremadura tomorrow, hopefully I'll have a chance to get a better hackergochi too.

OK, I know I've been very quiet on the blog for weeks now but I have my reasons and all will become clear, eventually. I'm principally waiting for the Lenny release before being able to start full discussions about certain ideas that have been consuming a lot of my time recently. I continue to do what I can to get Lenny out of the door because the wait is driving me nuts. (Fixed another RC bug yesterday, yippee!)

Anyway, something I can talk about is Emdebian Grip. Emdebian has always wanted to support "in-between" devices that don't need the full-on cross-build everything approach of the current Emdebian packages for ARM but don't quite have room for a full Debian install either. Until I had at least a basic package set available via the cross-building system, I didn't want to start working on an intermediate system, for reasons of simplicity mainly, but also because I knew that getting a cross-building package set would involve changes to the tools needed for the intermediary set.

Bah, I'm waffling again. For the background, see Emdebian, Squeeze, Grip and Crush.

The good news is that Emdebian Grip, once running, will take far, far less time (for everyone) than the full-on cross-building malarkey of Emdebian Crush. Grip can be 99.9% automated by breaking Debian into discrete package sets or 'components'.

Componentising Debian for Emdebian

Emdebian Grip is about a smaller Debian, nothing more. No changes in functionality, no changes in dependencies; only changes in final installation size and Packages.gz size. It means using the code from dpkg-cross to unpack packages, remove unwanted files and repack the .deb. Yes, yes, this can be done one package at a time at a trivial level. Emdebian Grip seeks to bring a standard method for this operation that will work for all packages on any architecture for any architecture from any suite, creating and maintaining repositories of "gripped" packages for use in devices like my Acer Aspire1 where you just don't need documentation, manpages and the rest. Squeeze is the obvious target release but any release supported by the current version of dpkg could be "gripped" too, like oldstable.

The componentisation comes in at the level of the Packages.gz file. If the device cannot realistically be expected to run CAD software or has no fancy graphics or other power-hungry hardware, why should packages that need such resources clutter up the apt-cache list? This data isn't just slowing down searching for packages you do want, it slows down every installation process (that bit about 'Reading database . . . . . '), it slows down the process of getting the updated cache data in the first place because there is more of it, it wastes space on your device in several different places when the whole point of Emdebian is that disc space is not cheap and it wastes your precious battery power if you need to do simple package management stuff away from the mains. What is needed is a mini-repository, a mini-mirror that just contains the parts of Debian that your machine needs - a "well-trimmed" mirror with sources and all dependencies satisfied, a mirror that can be automatically updated but which doesn't keep getting NEW packages that you won't use. New dependencies will be added, yes. New packages unrelated to what you currently use? Absolutely not. You could even keep a mini-mirror on your home network, updating and "gripping" while you sleep, because a mini-mirror should be a lot, lot smaller than a standard Debian mirror.

I now have a script that will do the work, what I've lacked is the idea on how to "feed" packages to the Gripper.

So, I'm going to be exploring this method:

1. An arbitrary package set - say the output of 'dpkg -l' from the target machine setup by an expert, or 'dpkg --get-selections' or just the packages from a normal debootstrap - the basic idea is to start with only the packages you absolutely must have for your target device.


2. A very small wrapper for some debootstrap code that downloads those .debs from the Debian mirrors - debootstrap can cope with choosing the mirror, the architecture, the suite etc. (already written, landing in Emdebian SVN soon).


3. Parse the list of packages (.debs) and use 'apt-cache showsrc' to convert into a list of source packages. (Script done).


4. 'apt-get source' to match the source package to the binary. At this point, we have what amounts to a partial mirror but with very well defined limits. There are other ways of doing the same thing so anything that gets you to this point is as equal as the steps above, as long as the .debs exist alongside the .dsc and all associated source package files. What you need is full control over the list of packages at this stage, we'll worry about dependencies later.


5. Simple scripting parses the .dsc to create suitable debian/files content (done).


6. dpkg-genchanges regenerates the .changes file - critical for keeping the Gripped packages with the correct source packages in the final Grip repository


7. Now run 'emgrip' against the .changes file to convert the .debs, update the .changes file with the new, smaller, .deb details and copy everything into a suitable incoming/ directory.


8. point your repository software (tested with reprepro) at the incoming/ directory and generate your Gripped repository - which only contains the packages you listed at the start, just smaller. reprepro has a neat way of allowing foobar-1.2.3-2 to be included even if foobar-1.2.3-1 does not exist in the repository, as long as foobar_1.2.3.orig.tar.gz exists somewhere that reprepro can find and even though foobar-1.2.3-2.dsc does not mention the .orig.tar.gz. Neat! (Tested and working.)


9. Run 'edos-debcheck' against the repository and "catch-up" with whatever packages are needed to make the repository as a whole "trimmed" (no unresolved dependencies).


10. Keep the update script running as a cron task to pull updated packages from incoming.debian.org, run the .changes through emgrip and update your repository.


11. Clean out the .debs, leave the source packages and restart the process for another architecture. (Crafty!)


12. Possibly, to support those who can afford to do a small amount of compilation on their Emdebian Grip device, add a second repository that provides the Build-Depends that may be missing from the initial package set - stuff like automake and pkgconfig. Depending on how that works out, it might not be many packages and it may be easier to fold them into the mini-mirror anyway.


13. Instead of 20,000 packages, I'm expecting something closer to 2,000, per mini-mirror.


14. The process of "gripping" a package also drops numerous control fields (like Homepage and VCS*), as well as Recommends, Suggests and Enhances as well as truncating the long description at a maximum of four lines - all in the name of reduced package data sizes.


15. The end result is that each mini-mirror contains a component package set for Emdebian Grip, using Debian packages. Mini-mirrors are still "well-trimmed" (no unresolved dependencies) because you get to choose which packages you include and if a particular package tries to bring in both Gtk and KDE dependencies, you can choose to omit that package and use a different one.


16. Whether Emdebian Grip will or should support dropping certain binary packages from the repository is a difficult issue - Emdebian Crush is probably the preferred route if you need to start optimising an Emdebian Grip mini-mirror that harshly.


17. Another wrapper will be needed to clean out cruft (old dependencies, outdated packages etc.) - that one may be harder to achieve by automation.

The first Emdebian Grip repositories will be for ARM, armel and i386, based on an XFCE environment package set.

"Gripping" all the packages in my /var/cache/apt/archives (about 1Gb) took about 30 minutes on a fast laptop that was doing other things at the time - the real packages will be "gripped" on the Emdebian server and for multiple architectures in sequence. It's certainly conceivable that any number of mini-mirrors can exist with any number of package sets (components), all with "gripped" packages. You can choose to "grip" .debs upon download from a normal Debian mirror but that wastes time if there is more than one device, it wastes space in /var/lib/apt/lists and /var/lib/dpkg/ and wastes bandwidth. For portable devices that you want to be using, not updating, I think that a mini-mirror is a worthwhile bonus.

Expect a 20% reduction in installation size (for more, you'll need the cross-building, functionality-modifying Emdebian Crush shebang) but a large decrease in the size of all the apt-cache data and dpkg status data. I'm hoping to push towards 30% reductions but most of these figures are guesswork right now.

And beyond the mini-mirrors? Debian-Installer, of course. I really need to set some time aside to prepare an Emdebian Grip installer using current D-I. Simple things like:

• Preferring ext2 instead of ext3 for solid state file systems,


• preferring XFCE to GNOME for the Desktop System,


• turning off Recommends before even the first package gets chosen - let alone installed,
  
• supporting machine:variant customisation scripting so that particular devices can be configured from first boot instead of having to rely on defaults or manual fixes via Wiki pages.

It really, really annoys me that I have to spend ages getting devices like the Acer Aspire1 working properly under Debian and that I have to take particular care during the installer to change the Debian default to something more sane for that kind of device. For Debian to be The Universal Operating System, the installer needs to be as smooth on an NSLU2* as on a standard desktop, a fast server and a cluster. This isn't just an architecture thing, there are big and small devices in all architectures, it needs to be a machine:variant thing. Mini-mirrors of single components are a start and part of the bigger answer is to support machine-specific configuration packages in these mini-mirrors. Packages that only exist in a specific mini-mirror and are specifically designed to setup a host of machine-specific settings, blacklist certain modules and enable others, set sensible settings in /etc/fstab and other places based on results from things like powertop, turning off features for some devices and turning them on for others. The whole point of such packages is that what they do would be insane on what would currently be considered a "typical Debian machine".

*(I'm not talking about an NSLU2 with an external hard drive here, I'm talking about fitting Emdebian Crush into the 6.4Mb of space on the NSLU2 itself so that external hard drives can be hot-plugged, as well as devices like the OpenMoko and iPAQs etc.)

Don't get me wrong, D-I does a fantastic job for the kinds of devices Debian currently supports but there are a host of other devices out there that need customised solutions where a mini-mirror and a machine-specific configuration package can remove huge amounts of hassle from the actual installation process. Emdebian Crush already uses machine:variant specific packages and the current Emdebian packages repository is, itself, a mini-mirror, albeit not well trimmed.

Let's make the most of "Debian Squeeze" - let's squeeze it onto more devices and make it even easier to get Debian onto anything.