After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

1. It is tightly tied to the version of perl which generated it.


2. It is, as expected, architecture-dependent


3. It is, unfortunately, very sensitive to the specific configuration used by the distribution itself

That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:

DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static 

That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :

/bin/bash debian/config.debian --full-version

The files to be copied out are:

• bitcount.h - architecture-specific (may be identical between similar architectures like armel & armhf)


• config.h - architecture & build specific, config.h.debug, config.h.static, config.h.shared.


• config.sh - architecture & build specific config.sh.debug, config.sh.static, config.sh.shared.


• uudmap.h - architecture-specific (may be identical between similar architectures like armel & armhf)

There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

1. compiler names (config.h.*)


2. architecture names (config.sh.*)


3. architecture-based paths (config.sh.*)

However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:

armhf: #define Netdb_host_t		const void * /**/

amd64: #define Netdb_host_t		char * /**/

I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:

dpkg-architecture -aarmel -c fakeroot debian/rules clean

That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

pyBit - cross-platform package building using AMQP

Message queues provide a simple way to create a distributed, cross-platform buildd toolkit to build packages using a collection of buildds, direct from various VCS clients. pyBit is intended to support rapidly evolving software collections and can support multiple VCS frontends and multiple build backends. Cross building is expected to be supported for some backends. The initial backend uses dpkg for Debian with subversion providing the source and sbuild doing the actual build.

pyBit includes support for cancelling selected builds and using multiple buildd clients per architecture, per platform and per suite.

Hooks are available or in development for subversion and git, other VCS hooks can be added. A RESTful web API provides live build reports and can generate build jobs for specific packages using particular VCS branches on selected architectures to support re-building packages at any point in the development process. Build history is stored using postgresql.

Other buildd systems can rebuild long lists of packages or build lots of binary packages from relatively slow moving source packages. PyBit exists to handle much more rapid software development across a wide range of platforms, VCS inputs and architectures. Buildd clients which are under-used can be tasked with building multiple suites or adding cross-build support. Buildd clients which are over-utilised are easily identified and adding new machines to an existing architecture / platform / suite pool should be easy. Hook activation automatically cancels any ongoing build for the same architecture, platform and suite to avoid wasting time on an interim version.

The emphasis in pyBit is to have fast builds with redundant clients, reliable reporting using a flexible and intuitive frontend. To this end, there is no need for a source package to be uploaded. Depending on the VCS hook in use, builds can happen every time a particular file is changed (e.g. debian/changelog for a native Debian package) or at every push (for a distributed VCS) or whatever is appropriate for a particular software collection. Builds are checked for available build-dependencies and packages re-queued if the build-dependencies are not yet met.

In the longer term, it may well be possible to use more than one server / database combination to support more builds and more platforms.

So far, we've got to a working model and tagged 0.1.0 as the first downloadable release. There is a lot more to do, especially adding more documentation, more VCS hooks, support for more VCS methods on the buildd clients and more buildd client scripts for platforms other than Debian. (The git hook and git source client are expected to let pyBit be self-buildable but a certain amount of configuration will be required for the server and each client which makes it not-quite self-hosting.)

pyBit concentrates on preparing collections of binary packages which can be used to build others, rather than trying to rebuild everything every time - this allows more rapid upstream software development and encourages modular, re-usable software. pyBit will also support rebuilding specific versions, architectures, suites or platforms via the RESTful web API. Access to this frontend can be controlled through any of the standard methods.

Components are loosely coupled via JSON encoded messages sent using rabbitMQ and curl. A new client can be added at any time and it will simply pick up buildd jobs from the relevant queue.

Development has now moved to Github and if anyone wants to look at new clients and new hooks, just contact the team by email or via IRC (#pybit on irc.oftc.net). Current development is based on Debian Squeeze 6.0.6 with backports and also on Debian Wheezy - more testing is welcome. Patches are also very welcome, pyBit is licensed under GPL2+.

There is no intrinsic reason why pyBit could not support any buildd platform capable of taking source from a known location and building a set of binaries. Packages are added to the queues whenever the hook is activated, so adding new packages to the collection is simply a case of triggering the existing hook.

Another interesting challenge for pyBit would be to trigger a hook on a new source code repository and just let it work through every package until everything is built. That probably won't work with the current 0.1.0 code but if that is what you'd like to work on, join the team.

Upgrading the local version of subversion against a large subversion repository has so far taken three DAYS. It goes through this multi-gigabyte repository in no perceivable order, it goes through every single directory in every single tag and every single branch and refuses to run in any sub-directory. (Even after clearing out some older tags, the tags/ directory is still nearly 8Gb and that doesn't include any binary files.) Each tag within tags/ is 500Mb and contains nearly 6000 sub-directories. Yes, it's large but svn upgrade should still be able to handle it without crippling every machine.

I can't image how svn is going to manage when the server finally gets upgraded. Probably be quicker to dump the entire repo and reimport it.

When I finally give up the will to proceed or simply need to use this LAPTOP for something else, I have to interrupt it with Ctrl-C at which point is starts all over again!

This isn't a slow machine, it's a i5 quad-core T410 with 8Gb of RAM and 1Tb of storage - and svn has made it crawl for days. The only way to pair down the working copy is to delete every tag and branch which means losing data like old build logs and old packages. The repository is this large because it's tracking the development of multiple commercial products which share common code but which also have numerous releases and release updates.

I can't even use svn st on this repo without this completing - I haven't been able to work on this repo since this started. Absolutely insane.

Pondering filing an RC bug on the basis of unjustifiable data loss. We have many machines at work with this repository checked out and if we ever migrate to Wheezy, it's going to mean a WEEK of lost work!!

subversion                    1.7.5-1

So, a warning for anyone else using subversion 1.7.5 with a very large repository: DELETE ALL TAGS and BRANCHES and any other directory anywhere in every single working copy tree on every machine which ever wants to use that copy again before even thinking about upgrading to 1.7.5.

The tags directories don't even need to be upgraded because we only use those to rebuild the code as it was in chroots.

Unspeakably furious about such a completely dumb tool being thrown into the mix DAYS before the Wheezy freeze. Now I have to kill it AGAIN just so that I can suspend the laptop. IDIOTS.

Not DebCamp this year, but I intend to get to more talks than I did in Bosnia and still get some work done on Emdebian Integration into Debian as well as working on as many RC bugs as I can manage during the week.

Possibly easy targets via UDD

A couple of my usual RC bug filter queries:


wheezy-and-sid, ignoring patch, pending, security, claimed and not in main. (483 bugs)


wheezy-or-sid, ignoring merged and done (987 bugs)

If you're in the habit of doing sudo debi at the end of a build, it's worth noting a complication with Multiarch.

Sometimes, it's tempting to rebuild (and install) a shared library with a few untested changes, however, if (like me), you have multiarch packages installed, this can cause a surprise:


dpkg: error processing ../libqof2_0.8.4-2_amd64.deb (--unpack):
trying to overwrite shared '/usr/share/doc/libqof2/changelog.Debian.gz', which is different from other instances of package libqof2:amd64


Bumping the changelog doesn't help:


dpkg: error processing libqof2:amd64 (--install):
package libqof2:amd64 0.8.4-3 cannot be configured because libqof2:armel is at a different version (0.8.4-2)
dpkg: error processing libqof2:armel (--install):
package libqof2:armel 0.8.4-2 cannot be configured because libqof2:amd64 is at a different version (0.8.4-3)


The fix for this is, of course, to remove libqof2:armel and all it's reverse dependencies and the packages can't be put back until you've built an armel version of your changes.

The way to backout the test change is a bit longwinded, depending on the complexity of your library:


sudo apt-get --reinstall install libqof2:amd64=0.8.4-2 libqof-dev:amd64=0.8.4-2 libqof2-backend-qsf:amd64=0.8.4-2 libqof2-backend-sqlite:amd64=0.8.4-2 libqof2-dbg:amd64=0.8.4-2 libqof2:armel=0.8.4-2


Hmm. I think a helper tool could be needed in the medium term here.

This also means that cross-building Emdebian Crush packages from a Wheezy / Wheezy+1 base is again looking as if it will need to happen inside a chroot, albeit with problems for the packages which are dependencies of the build tools (or the cross toolchain) itself, as the Crush packages will inevitably contain modified files. (For example, Emdebian Policy differs from Debian Policy by requiring - not forbidding - compression of debian/copyright.)

Cross-building packages which are intended to be binary compatible with Debian without conversion (including cross-built packages to use on top of Emdebian Grip) shouldn't be too affected. You just need to ensure that where the package exists in two or more versions, all installed architectures are built before any architecture is installed.

This is an additional burden compared to the world of dpkg-cross but it is key to how Multiarch allows cross-building to use sensible dependency resolution.

So, what I think we'll need is an enhancement to debi which can be passed an architecture (possibly an architecture list), debi then looks for both the native arch and the requested architecture(s), fails> if the .changes file for the extra architecture(s) isn't found or proceeds to install all packages for the native arch and arch-dependent packages for the second architecture in the same dpkg operation. As a further enhancement, debi might be able to check dpkg --print-foreign-architectures, check the output of $package:$arch against dpkg -l and either complain or automatically use the Multiarch support to look for the relevant .changes files. debi may even want to check the package (as it's working in the package top directory usually) for Multiarch support before accepting the architecture list option.

Think I need to do some reading of the debi source and see if a patch is workable...

I've been asked (or been criticised through indirect media) about why packages are removed and as I've just done quite a lot of removals at the Cambridge BSP (I've done some uploads too, it's not one-sided), I thought I'd explain my rationale in the hope that it will encourage others to remove more packages and fix those which warrant fixing. This, is an approximation of the kind of scoring I would use when assessing whether to fix a package or remove it. All bug counts are for the entire source package, including all bugs reported against binaries built from that source.

• R - Release critical bugs - count the bugs cumulatively - 1 point for 1 bug, 3 points for two (1+2), 6 points for 3 (1+2+3) etc.




• U - Uninstallable binary packages - extra points if a -dev package is involved as this will cause FTBFS in reverse dependencies. 1 point for every unrelated package affected. Add extra points if the uninstallable packages are reported as bugs rather than just as notices in the PTS. Add further points for each follow-up message to those bugs. (Filing or following up to a bug report is an indication of user impact as someone had to actually make a contribution.)




• B - Blocking other bugs - extra points for every RC bug which is blocking any other bug of any severity.




• B - Blocking transitions or release goals - extra points for every bug which is blocking a release transition. (Note this is not the same as being involved in the transition, it matters if the package is blocking the transition by making other packages involved in the transition unable to be rebuilt or migrate.)




• I - Interest - popcon is not much use here, "Interest" is a multiplier of the total of previous points. If the package is in testing, multiply the previous points total by the number of days the package has been waiting to enter testing beyond the normal limit. If it is blocking other packages from migrating, add the total number of excess days of all blocked packages. If the package is only in unstable, multiply the previous points total by the number of months the package has been waiting.




• S - Security - Count up the number of days the CVE has been waiting longer than a week, multiply that number of days by three and multiply the points so far by the result.




• H - Help - whether anyone has done any work at all, including the maintainer. Subtract 1 for any patches, 2 if the patches still apply and work. (It's not much for patches because if a patch exists and works, the excuses not to upload are less believable). Double the points so far for any working patches which have been ignored for more than a month (more than a week for a CVE or packages blocking transitions.

RUBBISH - easy to remember, maybe.

If someone comes up with a UDD query which can resolve the above as an algorithm, a) I'll buy them a beer at DebConf and b) it could be a useful addition to the PTS...

We needed a new package at work and once I'd written it, I realised that it could well be useful for others, so I did the ITP and the package is now in Debian unstable.

Ladder - Stepwise repository upgrade tool

Ladder creates a local repository of packages required

to upgrade a tarball of a chroot to a new milestone or

software release.

.

The repository can be signed and includes all specified

target packages and dependencies. The repository can then

be distributed and used to upgrade multiple devices in

sequence without needing network access.

The only dependencies are some simple perl modules, reprepro for the repository, apt and gnupg for the signing. None of those need to be particularly recent versions, so once ladder has had a little time in testing, I will be doing a backport to Squeeze. Lenny is a little less likely but eminently possible if someone specifically asks for it. Not that it particularly needs a backport mind, if you want to run ladder on some other Debian-based system, it's simply a case of installing the version in unstable. reprepro was in Lenny at a suitable version for what ladder requires, so any system remotely recent should be able to run ladder without changes.

Normally, whenever you need to upgrade a device, you need to get that device onto a network, get access over the serial port or some form of pre-configured network connection, run the commands and clean up afterwards. OK, now do that again... oh and here are another 40 or 100 devices which all need precisely the same thing done and they all have to be shipped to paying customers today. What does any geek do? Script it.

Ladder is part of that process. There would need to be some scripting / programming support on the devices concerned but ladder makes it easier to put that support in place at the design stage and then automate the actual update without the device needing to get onto a network and, potentially, allowing you to decide how to offer the upgrade (automatic, user involvement, engineers only etc.) Once that is implemented, the device is simply pointed at a locally mounted filesystem which contains the ladder step for the migration. Each step is a SecureApt signed repository which is available at a deterministic location which can be easily scripted and which contains only the packages necessary (with dependencies) to upgrade any device installed at Milestone A to Milestone B. Nice and small, only the stuff you need, whatever architecture you like and if you need to migrate from Milestone A to Milestone D, then you create a few steps on the ladder and automate the migration from A to B to C to D. All you need is enough space on whatever local media you need to use to plug into the devices. Then copy the media enough times, start the interface on each device and do something useful whilst the upgrade happens. Then, when the next lot need to be upgraded, you already have the media... Oh and ladder does not need to run on the same architecture as the devices - all package handling is entirely architecture-neutral. So create the ladder steps and media on your fast x86 machines and then let the slower SSD devices take their own sweet time via automation. Simple - maybe, hopefully.

I'm expressly talking about devices here because this is where the original requirement arose - embedded devices which don't have direct / automatic / accessible network ports or even a working network configuration preset. These aren't new images to flash onto the device either, these are apt repositories, so only the stuff which needs to be changed is changed which saves a lot of time writing to slow SSD storage.

If the mechanism sounds familiar, yes, it's using the same principles as xapt, multistrap and emdebian-grip but I thought it could be useful, so I generalised the interface (a bit) and uploaded.

Let me know via the BTS if you need tweaks to the support. Ladder isn't particularly about getting desktop machines from Debian Potato to Debian Wheezy because you need to identify a target package to be the top point of the dependency chain which gets into the step repository and all devices should start with the same packages at the same versions. Normally, the kind of devices I use at work have a single top level application which is the sole user interface and which directly or indirectly depends on all the other software required for that device. Whilst the target package value could take a few carefully selected packages instead of just one and there is support for an extra packages field, it's not intended / expected to be useful for desktops. This is for use with multiple devices which all have the same package selection, are at the same milestone at the start and which all need to migrate smoothly to whatever milestone is intended - one step at a time. Ladder could be used to migrate between Debian releases and I've included example config files to support that, but the principle usage is with internal / proprietary repositories based on a common Debian stable platform where the user interface software is managed via identifiable milestones and where users have no direct control over installing packages. It's a production / manufacturing support tool more than a user / admin support tool. If you want to manage servers and desktops which allow users to install (or request to be installed) arbitrary packages, there are plenty of tools which already support this and are used by DSA in Debian for exactly these tasks. Ladder is not puppet but Linp didn't sound like a good name...

Ladder step repositories only include the binary packages, not sources - so if the migration is not going to be done in-house and you're expecting to distribute these steps to users somehow, ensure that the original source repository is available online for the Debian / free software packages concerned. If you can't do that (because the source is proprietary), you possibly shouldn't be distributing the binaries as a user update mechanism in the first place because each step will include packages from the core Debian system which need to have the sources available when distributed.

Inevitably, the ladder source package already contains two POT files for translations of the runtime messages and the POD based manpage. I expect to need to expand / clarify the manpage in due course, so don't rush to translate it yet as it's likely to change.

With dpkg from experimental and the new zlib upload in unstable, I've now got a partial Multi-Arch install. There are more packages necessary, particularly related to how -dev packages can exist and how a cross compiler gets built/installed in Multi-Arch world. One bug #659588 in libglib2.0-0 so far but that's quite good seeing as it's been all but impossible to test the Multi-Arch changes in existing packages until now.

$ dpkg --print-foreign-architectures

armel

i386

$ dpkg -l | cut -c -80 | grep armel|grep -v cross

ii  gcc-4.6-base:armel                   4.6.2-14                     

ii  libc6:armel                          2.13-26                    

ii  libdatrie1:armel                     0.2.5-3                

ii  libffi5:armel                        3.0.10-3                 

ii  libgcc1:armel                        1:4.6.2-14                 

ii  libgmp10:armel                       2:5.0.4+dfsg-1         

ii  libgmpxx4ldbl:armel                  2:5.0.4+dfsg-1        

ii  libgomp1:armel                       4.6.2-14         

ii  libmpc-dev:armel                     0.9-4              

ii  libmpc2:armel                        0.9-4          

ii  libmpfr-dev:armel                    3.1.0-3         

ii  libmpfr4:armel                       3.1.0-3       

ii  libpcre3:armel                       8.12-4                

ii  libpixman-1-0:armel                  0.24.4-1    

ii  libpng12-0:armel                     1.2.46-4     

ii  libpopt0:armel                       1.16-3        

ii  libpwl5:armel                        0.11.2-6    

ii  libselinux1:armel                    2.1.0-4.1      

ii  libstdc++6:armel                     4.6.2-14       

ii  libthai0:armel                       0.1.16-3     

ii  linux-libc-dev:armel                 3.2.4-1         

ii  zlib1g:armel                         1:1.2.6.dfsg-1   

There are more packages which can be installed on amd64 for i386 as #659588 only affects Multi-Arch versions which cannot execute compiled binaries from the foreign architecture.

Further progress, inside a test chroot, involves using gcc-4.7 from experimental but even then, libc6-dev is not installable as a Multi-Arch package. That's the current blocker for toolchain stuff.

ii  gcc-4.7-base:i386               4.7-20120210-1

ii  libgcc1:i386                    1:4.7-20120210-1

Once we have libc6-dev:armel installable on i386 and amd64, work on cross-building Emdebian can be considered again. It's been a long time.