Emdebian Grip automated dependency resolution

The script isn't fully automated yet, I'm running it manually and watching for errors, but I do now have a simple script (~100 lines of perl) which runs edos-debcheck against each of the supported Emdebian architectures for unstable-grip and picks out those packages which are missing from the subset of packages which Emdebian monitors for Grip.

I was hoping to make these scripts into a normal Debian package but there are some limitations in that the scripts need to assume various pieces of Debian infrastructure. e.g. I started off using rsync to pull in the Packages files (along with Release and Sources) because apt has annoying behaviours with multiple architectures. (Emdebian Grip processes 7 architectures simultaneously on blavet.debian.org). Peter Palfrader kindly arranged for a local mirror to be available and now the entire Packages processing can be done using symlinks. Much, much better.

Actual package movement data comes directly from dak via projectb - something else which makes packaging these scripts for use on non debian.org boxes a bit hard.

Once again, I am indebted to the EDOS team (please, please keep working on edos-debcheck - it hasn't had uploads recently) because I tried to get this working with germinate but failed. (Colin - I'll be looking to borrow some of your time to work out what was going wrong.)

The dependency resolution script now means that I can meet release team requirements that when foo is updated in sid to depend on libbar2 but Emdebian Grip only has libbar1, the scripts will automatically pick up libbar2 and add it to Emdebian Grip. The current sid version gets pulled in and (the theory goes) will migrate into testing-grip as an almost automatically valid candidate. Processing of the dependency can happen within hours of the updated version of foo arriving in Emdebian Grip but as long as it happens within a few days I think everyone will be fine with it.

I've got a sync script too which can periodically run across the entire suite and check for packages which have slipped the net. I'll have to see how often that needs to run but once a week doesn't sound too painful.

Processing for Emdebian Grip is very fast. Most of the time is taken uploading:

2012-01-23 22:24:42 add sid-grip deb main armhf libfltk-images1.3 1.3.0-5em1

2012-01-23 22:24:52 add sid-grip deb main i386 libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:07 add sid-grip deb main mips libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:25 add sid-grip deb main powerpc libfltk-images1.3 1.3.0-5em1

2012-01-23 22:26:00 add sid-grip deb main amd64 libstdc++6-4.5-dev 4.5.3-12em1

2012-01-23 22:26:34 add sid-grip deb main armel libstdc++6-4.5-dev 4.5.3-12em1

(It was a lot longer, comparatively, when I was having to rsync and then dget instead of just read from the local filesystem.)

I've also been working on an init script which will gradually work through the subset of packages by order of Priority - approximately what a new architecture (like armhf) would do. Note that Emdebian Grip supports armhf already. The only slight concern is whether the init script needs to be throttled back to not flood the queues with hundreds of uploads an hour (which it could end up doing, at least for the first 24 hours).

Logs of package movements are available: http://www.emdebian.org/grip/logs.php

Packages can be searched too: http://www.emdebian.org/grip/search.php

More on Emdebian Integration on the Debian wiki: http://wiki.debian.org/EmdebianIntegration

Now all I need is a fix for #655919 FTP team?