Codehelp blog

pybit 1.0.0 - distributed, scalable builds direct from VCS or archives

nospam@example.com (Neil Williams) — 2013-05-19T00:02:28Z

PyBit is a distributed build system able to build packages in response to VCS commits or other triggers, across multiple architectures, multiple clients and multiple build environments with automated uploads to a nominated repository.

Support is included in 1.0.0 for building Debian packages using sbuild in response to subversion commits or changes in debian-devel-changes@lists.debian.org (by using apt as a version control handler) for any architecture and build environment which sbuild can support. There is also an example git commit template. Pybit has been designed to be fully extensible, so support for RPM or other package formats can be added as well as other version control handlers, other build environments and other architectures. Pybit is also scalable, when one type of client is struggling with the workload, another machine of the same architecture can be added to the pool to share the load. Pybit can also build a package for any number of architectures and build environments at the same time. The Pybit web interface provides an at-a-glance summary of all current builds as well as options to blacklist certain combinations, cancel and retry specific jobs and add monitor each pybit client. Current use cases include:

Rapidly changing VCS - one or more subversion repositories with lots of Debian packages, built automatically for any number of build environments and architectures every time the debian/changelog is modified. Clean chroot builds provide continuous integration testing of the every package.

Rebuilding the archive with different compilers or flags - a dedicated email account subscribed to debian-devel-changes@lists.debian.org feeding messages through procmail to the changes-debian hook, passing build requests to the apt handler to rebuild each package in your own sbuild chroots, using whatever environments, suites and build options can be configured within those chroots.

something else we haven't thought of yet ... there is scope for a lot more hooks, package formats, chroot tools and handler plugins.

arrived in Debian unstable

https://freecode.com/projects/pybit

http://nicholasdavidson.github.io/pybit/

https://github.com/nicholasdavidson/pybit

Update on klibc for AArch64

nospam@example.com (Neil Williams) — 2013-02-08T15:17:22Z

klibc_2.0.1-3.2_arm64.changes

I've now completed the first patch for klibc 2.0.1 for arm64 support - it's completely untested and likely to break things, so it's up for review not usage. Those who are brave enough to look at the patch, please report issues via the github issue tracker.

35 files changed, 509 insertions(+), 19 deletions(-)

21 new C files to replace removed syscalls, most of which simply borrow from glibc and provide a general purpose call (like open) instead of the actual syscall (openat).

Next is to test the build and look into what is necessary to push the patch to the current 2.0.3 upstream.

Hard drive death

nospam@example.com (Neil Williams) — 2013-01-25T10:14:09Z

My 1Tb laptop drive started misbehaving a few weeks ago, just spending a lot of time spinning when it should have been reading frequently changed files (like the browser cache). I was tempted to blame the browser at that point as no other packages appeared affected.

Wednesday night, a routine package upgrade on unstable brought in a bunch of qt4 updates which I wanted and a virtual box update that I didn't see much point in delaying ... 15 minutes later I couldn't work out why the virtualbox dkms task was still running, spotted it spinning in depmod and found some alarming messages in dmesg about short reads and errors reading from the hard drive. Hmm. The hard drive was out of control at this point, it quickly became apparent that no disc access was going to be possible and I couldn't get to a terminal to kill the current tasks, so I killed the power. The fsck which followed reboot showed more of the errors I'd seen in dmesg and fell back to the manual intervention stage. A few hours of confirming attempts to fix the errors, fsck finally finished. A short amount of usage showed that although fsck had finished, the drive was not happy and was starting to give short reads on other parts of the filesystem, resulting in ~40% of the filesystem appearing to be read-only when the rest was read-write. Somehow, I didn't think this was a welcome feature as the areas affected appeared to be quite random.

The drive in question was a replacement for the original 300Gb drive supplied with the ThinkPad, so a quick bit of switching of drives into a caddy and I could rsync my data off the large drive onto the smaller one. The rsync itself took a lot longer than it should have done because it got lots of short reads too. (Principally in /lib/modules/3.2.0-4/ and in the browser cache directories which had been the original symptom as well as most other places where one could have expected processes to have open files when the drive failed).

Now the 1Tb drive was a pig to fit into the Thinkpad originally because it was too big for the bay but I fitted it anyway. Yes, that was probably a mistake. It certainly meant that in order to fit it I had to not put the drive into the useful caddy provided by Lenovo which makes removal of the drive simple. Indeed the drive was wedged into the bay so tightly that it wasn't going to come out with normal levels of persuasion. This probably contributed to the failure of the drive, so live and learn.

With help from Andy Simpkins (it's always handy to have a hardware engineer on hand at times like this), the keyboard was lifted out, the case was dismantled and just enough room was made to get a screwdriver in behind the sata drive and lever it out of the bay. OK, rebuild laptop, put replacement drive into the caddy (because the smaller capacity drive is also a lot smaller in height than the 1Tb and therefore has plenty of clearance between it and the bay) and move on to the software recovery stage.

Hint: if this happens again, before turning off the broken system for the last time, just remember to download a recent Debian ISO to a USB stick - it saves having to ask someone else or find another machine to do the download. (Thanks Andy...)

OK, so after the usual complaints on reboot that there was no operating system, F12 got the boot order menu up and I was in Debian Installer Rescue Mode. Reinstalling grub failed initially for a few reasons:

I'd deliberately not copied /dev /proc and /sys from the old harddrive but I'd also forgotten to create empty ones on the new drive...
/etc/fstab helpfully referred to / as a UUID which no longer applied, so that had to be edited
grub was confused and couldn't find the root filesystem

A few iterations later, I had a working /dev directory inside the /target chroot, bind mounted from the /dev outside the chroot, I was able to mount proc and sys, so grub was finally happy to reinstall itself and then update the initramfs setup for the new drive.

Reboot, another fsck, all appeared well. I was able to login via the terminal but not in X. Hmmm. Stop xdm, startx manually from the terminal, problems with /tmp/ - permission denied. Oops. Yes, it does help to create /tmp with the right permissions....

The final stage was to complete the dpkg --configure -a from the original failure. I've yet to reinstall the 3.2.0-4 kernel, so I'm back on 3.2.0-3 but that's OK.

So now I'm back on the original drive, albeit temporarily without any swap whatsoever (because I didn't partition the replacement drive to create /dev/sda5 before doing the copy) and now I remember the second reason I wanted to replace the original drive with the 1Tb drive - the original drive is as NOISY as hell. The whole edge of the laptop vibrates constantly, to the point that I can feel the vibration under the keys as I type. It's not that the drive is loose in the bay, it's just a constant vibration.

But, I have kept all my data and I have a usable laptop for the BSP this weekend. I will be looking at an SSD drive to replace this one though and having also found my old Acer laptop with power supply, I can now reference this entry when I transfer the system a second time.

Status of klibc on AArch64

nospam@example.com (Neil Williams) — 2013-01-13T10:02:27Z

With significant assistance from Steve McIntyre and some judicious delving into the ARM Information Centre, I've now got the assembly portions of klibc sorted (but untested) for AArch64 (arm64).

Andy & I started by copying the old ARM support as a new directory and one of the final steps was to remove a whole bunch of legacy code from the days before Thumb and all the #ifdef lines which went with it. Some files disappeared entirely. setjmp.S was the largest amount of work as the load&store multiple support of ARMv7 has gone in ARMv8, so the stmia mnemonic had to be expanded to multiple stp calls but that makes it more explicit, so it's not a bad thing.

Steve & I borrowed from the glibc AArch64 code from upstream by Marcus Shawcroft, simplifying the glibc macros for klibc and also got clues about what extra registers needed handling compared to ARMv7.

I've now got to look at some traditional cross-compilation issues because the Linaro AArch64 toolchain doesn't install to typical Debian cross-building paths and the build now moves past the AArch64-specific assembly and fails later when the C code gets the wrong include path and ends up including /usr/include/i386-linux-gnu/asm/byteorder.h with predictable results.

If someone has an AArch64 toolchain already set up, feel free to clone my modified klibc tree and let me know if there are subsequent build errors. Of course, if you fancy testing a build in the Foundation Model for ARMv8, that would be good too! (Report issues via github.)

Whilst I'm sorting out the toolchain, I've also been updating perl-cross-debian (which has also seen some more upstream testing and improvement).

perl-cross-debian updates

nospam@example.com (Neil Williams) — 2013-01-11T22:08:00Z

Working with perl upstream, we're getting closer to a fully cross-built upstream perl without needing the external perl installation. The branch (which is also available here with a few of my changes) now builds a host miniperl, cross-builds the rest of perl and almost gets through the rest of the build by using the host miniperl to handle the extensions up as far as XS::Typemap:



make[1]: Leaving directory `/home/neil/code/debian/src/perl/cross/git/perl/ext/XS-APItest'

./miniperl -Ilib make_ext.pl lib/auto/XS/Typemap/Typemap.so MAKE=make LIBPERL_A=libperl.a LINKTYPE=dynamic

	Making XS::Typemap (all)

make[1]: Entering directory `/home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

make[1]: Leaving directory `/home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

Making all in ext/XS-Typemap

make all PERL_CORE=1 LIBPERL_A=libperl.a LINKTYPE=dynamic

make[1]: Entering directory `/home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

make[1]: Leaving directory `/home/neil/code/debian/src/perl/cross/git/perl/ext/XS-Typemap'

./perl -f -Ilib pod/buildtoc -q

Can't load module Digest::MD5, dynamic loading not available in this perl.

  (You may need to build a new perl executable which either supports

  dynamic loading or has the Digest::MD5 module statically linked into it.)

 at Porting/pod_lib.pl line 4.

Compilation failed in require at Porting/pod_lib.pl line 4.

BEGIN failed--compilation aborted at Porting/pod_lib.pl line 4.

Compilation failed in require at pod/buildtoc line 17.

BEGIN failed--compilation aborted at pod/buildtoc line 18.

make: *** [pod/perltoc.pod] Error 2

Note the change from ./miniperl (which is itself a bug as it should be ./host/miniperl) to ./perl which is, naturally, an armel binary. It's also copied into the local directory, replacing the system perl if I copy it in.

So, more to do, but at least it gets this far.

Improved support for the extensions should also make it easier to clean up the current Debian cross-build diff which is the remaining bit of awkwardness / kludge.

bootstrapping arm64

nospam@example.com (Neil Williams) — 2013-01-06T18:34:01Z

I'm still working on perl-cross-debian (just uploaded 0.0.2) and there's more to do on that with upstream but part of the reason to work on perl cross-building is to do what I can to help with the ARM64 port.

So, I went back to Wookey's talk at DebConf12 which leads to the current list of cross-build results for arm64 and started through the list.

coreutils is listed as failing but that was an archive error (MD5sum hash mismatch), so that just needs a retry. I don't have access to that buildd, yet, so nothing I can do there.

Next on the list (excluding those just waiting for build-deps) was klibc.



  aarch64-linux-gnu-gcc -Wp,-MD,usr/klibc/.__static_init.o.d  -nostdinc -iwithprefix 

include -I/«PKGBUILDDIR»/usr/include/arch/x86_64 -Iusr/include/arch/x86_64 

-I/«PKGBUILDDIR»/usr/include/bits64 

-Iusr/include/bits64 -I/«PKGBUILDDIR»/usr/klibc/../include 

-Iusr/klibc/../include -I/«PKGBUILDDIR»/usr/include -Iusr/include 

-I/«PKGBUILDDIR»/linux/include -Ilinux/include 

-I/«PKGBUILDDIR»/linux/arch/x86/include -Ilinux/arch/x86/include 

-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=64 -fno-stack-protector -fwrapv -m64 -Os 

-fno-asynchronous-unwind-tables -fomit-frame-pointer -falign-functions=1 -falign-jumps=1 -falign-loops=1 

-W -Wall -Wno-sign-compare -Wno-unused-parameter -c -o usr/klibc/__static_init.o usr/klibc/__static_init.c

aarch64-linux-gnu-gcc: error: unrecognized command line option '-m64'

make[4]: *** [usr/klibc/__static_init.o] Error 1

Turns out that this is a build failure I understood, at least initially. A little digging and a trivial patch was begun:



--- a/usr/klibc/arch/arm/MCONFIG

+++ b/usr/klibc/arch/arm/MCONFIG

@@ -30,6 +30,13 @@

 ifeq ($(CONFIG_AEABI),y)

 KLIBCREQFLAGS += -mabi=aapcs-linux -mno-thumb-interwork

 else

+# aarch64

+ifeq ($(CONFIG_AARCH64),y)

+KLIBCREQFLAGS +=

+KLIBCOPTFLAGS += -mgeneral-regs-only

+else

 KLIBCREQFLAGS += -mabi=apcs-gnu -mno-thumb-interwork

 endif

 endif

+endif

+

Alongside a trivial change to debian/rules



ifeq ($(DEB_HOST_ARCH),arm64)

DEB_MAKE_ENVVARS := ARCH=arm CONFIG_AARCH64=y CPU_ARCH=armv8-a CPU_TUNE=generic

endif

OK, then things get a bit more awkward,



....

code/debian/cross/klibc/klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include -D__KLIBC__=2 

-D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os 

-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter 

-c -o ../foo.o usr/klibc/arch/arm/crt0.S

usr/klibc/arch/arm/crt0.S: Assembler messages:

usr/klibc/arch/arm/crt0.S:19: Error: operand 1 should be an integer register -- `mov r0,sp'

usr/klibc/arch/arm/crt0.S:20: Error: operand 1 should be an integer register -- `mov r1,#0'

Hmm. Assembly, well, yes, I've done assembly before, I know what mov should normally do, sp is likely to be the stack pointer .... where's my AARCH64 assembly PDF again... PRD03-GENC-010197 ...

OK, so maybe the r0 and r1 should be x0 and x1, hmm, that at least doesn't raise assembly errors. So a tentative change:



--- a/usr/klibc/arch/arm/crt0.S

+++ b/usr/klibc/arch/arm/crt0.S

@@ -15,9 +15,13 @@

 #ifdef __thumb__

 	.thumb_func

 #endif

-

+#ifdef __aarch64__

+_start: mov x0, sp

+	mov x1, #0

+	bl __libc_init

+#else

 _start:	mov	r0, sp

 	mov	r1, #0

 	bl	__libc_init

-

+#endif

 	.size _start,.-_start

Next stage, however, leaves me quite a bit more lost:



....

klibc-2.0.1/linux/arch/arm/include -Ilinux/arch/arm/include 

-D__KLIBC__=2 -D__KLIBC_MINOR__=0 -D_BITSIZE=32 -fno-stack-protector -fwrapv -fno-exceptions -Os 

-march=armv8-a -mtune=generic -mgeneral-regs-only -W -Wall -Wno-sign-compare -Wno-unused-parameter 

-c -o usr/klibc/arch/arm/setjmp.o usr/klibc/arch/arm/setjmp.S

usr/klibc/arch/arm/setjmp.S: Assembler messages:

usr/klibc/arch/arm/setjmp.S:32: Error: unknown mnemonic `stmia' -- `stmia r0,{r4,r5,r6,r7,r8,r9,r10,fp,sp,lr}'

usr/klibc/arch/arm/setjmp.S:33: Error: operand 1 should be an integer register -- `mov r0,#0'

usr/klibc/arch/arm/setjmp.S:34: Error: operand 1 should be an integer register -- `mov pc,lr'

usr/klibc/arch/arm/setjmp.S:42: Error: unknown mnemonic `ldmia' -- `ldmia r0,{r4,r5,r6,r7,r8,r9,r10,fp,sp,lr}'

usr/klibc/arch/arm/setjmp.S:43: Error: unknown mnemonic `movs' -- `movs r0,r1'

usr/klibc/arch/arm/setjmp.S:44: Error: unknown mnemonic `moveq' -- `moveq r0,#1'

usr/klibc/arch/arm/setjmp.S:45: Error: operand 1 should be an integer register -- `mov pc,lr'

make[5]: *** [usr/klibc/arch/arm/setjmp.o] Error 1

So now I'm out of my depth of AARCH64 assembly (apart from the recurrence of mov r0 vs mov x0 etc.). If the above is useful then maybe someone can work out what is wrong with setjmp.S or whether AARCH64 just means that klibc needs to gain a arch/arm64/ directory and not try to duplicate each entire assembly block within #ifdef clauses.

I don't really know where else to put an incomplete investigation like this, so it's here for anyone to find.

(Oh, and if you're reading those arm64 cross-build logs, then a few hundred occurrences of ldconfig: /usr/lib/aarch64-linux-gnu/*.so is for unknown machine 183. in every build log (success or fail) is apparently entirely normal until more packages get fixed. Really slows down scanning the build log.

I may try busybox or libusb next. libusb looks like a classic "you might have told me to cross-compile but I'm going to use g++ anyway because I know best" cross-building problem, indicative of yet another BDBS. sigh.

Resources:
Getting started with 64-bit ARM development
ARMv8 images for developers
AArch64 for everyone, Marcin Juszkiewicz
Howto/HelloAarch64 - Linaro wiki
AArch64 gcc options

perl-cross-debian is ready!

nospam@example.com (Neil Williams) — 2012-12-13T12:34:05Z

This is the listing for my local cross-build-only repository for perl:



perl | 5.14.2-16 | unstable | armel, source

perl | 5.16.2-1 | experimental | armel, source

Prior to this, 5.14.2-15 also cross-built.

I've just pushed the update which fixes 5.16 from current Debian experimental.

This means that I'm ready to push perl-cross-debian into experimental via NEW. Whilst the package is in NEW, I will be approaching perl upstream about the necessary changes for Makefile.SH and updating the existing bug reports #285559 and #633884 with the necessary changes for debian/rules.

In the meantime, the changes for Makefile.SH and debian/rules exist within the perl-cross-debian source code - the patch for 5.14.2 is the same as I used for 5.16.2 and I don't see a need, yet, for this to be any different with current perl upstream. Equally, the patch for debian/rules works equally well for 5.14.2 and 5.16.2. All the version-specific (and architecture-specific) data lives in perl-cross-debian.

So it's time to tag perl-cross-debian 0.0.1 and upload to ftp-master as a native package aimed at experimental.

What's left to do? TESTING!

I've only tested with armel using the Emdebian cross-building toolchains from Squeeze using the old dpkg-cross style cross-dependency installation paths. There is outline code for armhf and arm64 but these need testing. The code also needs testing using the latest MultiArch cross-building toolchains. This should be a simple matter of checking if the dpkg-cross style paths exist and looking for MultiArch if not.

Right now, all of this is "worksforme" grade. It needs others to have a go and file bugs. Until the package is through NEW, feel free to use the issue tracker on the perl-cross-debian github site.

Please read through the documentation in the source code and the manpages in the package (xml in the source code) and tell me if some of it isn't clear.

perl-cross-debian update

nospam@example.com (Neil Williams) — 2012-11-25T16:30:13Z

Long term maintenance of cross-build support for the Debian configuration of perl has now gained some code at github and an ITP: #694326 for Debian.

There's some working code for perl 5.14 and initial work on 5.16 (which isn't complete yet).

This will dramatically simplify the patch for #633884 and provide a base for getting another part of that patch into upstream (Makefile.SH). (Thanks to Steve McIntyre & Peter Pearse for the body of the patch itself.)



ifneq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))



define variant

$(if $(findstring static,$1),static,$(if $(findstring debug,$1),debug,shared))

endef



define cross-config

    /usr/bin/perl-cross-debian --variant $(call variant,$@)

    perl -Ilib  make_patchnum.pl

endef

endif

So the config*variant files will live in /usr/share/perl-cross-debian/$arch/$perl_version/

Long term maintenance of perl cross-build support in Debian

nospam@example.com (Neil Williams) — 2012-11-18T17:29:48Z

After prompts from Wookey and Steve McIntyre, I decided to look at #285559 and #633884 for perl cross-build support and then port that support forward to the current perl in Wheezy and on to the version of perl currently in experimental. The first patch is for perl 5.8, the second for perl 5.12, neither of which is available currently in Debian. snapshot.debian.org provided the 5.12 source but then that no longer cross-builds with the patch.

The problem, as with any cross build, is that the build must avoid trying to execute binaries compiled within the build to achieve the test results required by ./configure (or in the case of perl, Configure). dpkg-cross has one collection of cache values but the intention was always to migrate the package-specific support data into the packages themselves and keep the architecture-specific data in dpkg-cross or dpkg-dev. Therefore, the approach taken in #633884 would be correct, if only there was a way of ensuring that the cached values remain in sync with the relevant Debian package.

I'll note here that I am aware of other ways of cross-building perl, this is particularly concerned with cross-building the Debian configuration of perl as a Debian package and using Debian or Emdebian cross-compilers. After all, the objective is to support bootstrapping Debian onto new architectures. However, I fully expect this to be just as usable with Ubuntu packages of perl compiled with, e.g. Linaro cross-compilers but I haven't yet looked at the differences between perl in Debian vs Ubuntu in any detail.

I've just got perl 5.14.2 cross-building for armel using the Emdebian gcc-4.4 cross-compiler (4.4.5-8) on a Debian sid amd64 machine without errors (it needs testing, which I'll look at later), so now is the time to document how it is done and what needs to be fixed. I've already discussed part of this with the current perl maintainers in Debian and, subject to just how the update mechanism works, have outline approval for pushing these changes into the Debian package and working with upstream where appropriate. The cache data itself might live in a separate source package which will use a strict dependency on perl to ensure that it remains in sync with the version which it can cross-build. Alternatively, if I can correctly partition the cache data between architecture-specific (and therefore generated from the existing files) and package_$version specific, then it may be possible to push a much smaller patch into the Debian perl package. This would start with some common data, calculate the arch-specific data and then look for some version-specific data, gleaned from Debian porter boxes whilst the version is in Debian experimental.

The key point is that I've offered to provide this support for the long term, ensuring that we don't end up with future stable releases of Debian having a perl package which cannot be cross-built. (To achieve that, we will also end up with versions of perl in Debian testing which also cross-build.)

This cross-build is still using dpkg-cross paths, not MultiArch paths, and this will need to be changed eventually. (e.g. by the source package providing two binaries, one which uses MultiArch and one which expects dpkg-cross paths.) The changes include patches for the upstream Makefile.SH, debian/rules and the cache data itself. Depending on where the cache data finally lives, the new support might or might not use the upstream Cross/ directory as the current contents date from the Zaurus support and don't appear to be that useful for current versions of perl.

The cache data itself has several problems:

It is tightly tied to the version of perl which generated it.

It is, as expected, architecture-dependent

It is, unfortunately, very sensitive to the specific configuration used by the distribution itself

That last point is important because it means that the cache data is not useful upstream as a block. It also means that generating the cache data for a specific Debian package means running the generation code on the native architecture with all of the Debian build-dependencies installed for the full perl build. This is going to complicate the use of this method for new architectures like arm64.

My objective for the long term maintenance of this code is to create sufficient data that a new architecture can be bootstrapped by judicious use of some form of template. Quite how that works out, only time will tell. I expect that this will involve isolating the data which is truly architecture-specific which doesn't change between perl versions from the data which is related to the tests for build-dependencies which does change between perl versions and then work out how to deal with any remainder. A new architecture for a specific perl version should then just be a case of populating the arch-specific data such as the size of a pointer/char and the format specifiers for long long etc. alongside the existing (and correct) data for the current version of perl.

Generating the cache data natively

The perl build repeats twice (three builds in total) and each build provides and requires slightly different cache data - static, debug and shared. Therefore, the maintenance code will need to provide a script which can run the correct configuration step for each mode, copy out the cache data for each one and clean up. The script will need to run inside a buildd chroot on a porter box (I'm looking at using abel.debian.org and harris.debian.org for this work so far) so that the derived data matches what the corresponding Debian native build would use. The data then needs slight modification - typically to replace the absolute paths with PERL_BUILD_DIR. It may also be necessary to change the value of cc, ranlib and other compiler-related values to the relevant cross-compiler executables. That should be possible to arrange within the build of the cache data support package itself, allowing new cache files to be dropped in directly from the porter box.

The configuration step may need to be optimised within debian/rules of perl itself as it currently proceeds on from the bare configuration to do some actual building but I need to compare the data to see if a bare config is modified later. The test step can be omitted already. Each step is performed as:



DEB_BUILD_OPTIONS="nocheck" fakeroot debian/rules perl.static

That is repeated for perl.debug and libperl.so.$(VERSION) where $VERSION comes from :



/bin/bash debian/config.debian --full-version

The files to be copied out are:

bitcount.h - architecture-specific (may be identical between similar architectures like armel & armhf)

config.h - architecture & build specific, config.h.debug, config.h.static, config.h.shared.

config.sh - architecture & build specific config.sh.debug, config.sh.static, config.sh.shared.

uudmap.h - architecture-specific (may be identical between similar architectures like armel & armhf)

There is a lot of scope for templating of some form here, e.g. config.h.debug is 4,686 lines long but only 41 of those lines differ between amd64 and armhf for the same version of perl (and some of those can be identified from existing architecture-specific constants) which should make for a much smaller patch.

Architecture-specific cache data for perl

So far, the existing patches only deal with armel and armhf. If I compare the differences between armel & armhf, it comes down to:

compiler names (config.h.*)

architecture names (config.sh.*)

architecture-based paths (config.sh.*)

However, comparing armel and armhf doesn't provide sufficient info for deriving arm64 or mips etc. Comparing the same versions for armhf and amd64 shows the range of differences more clearly. Typical architecture differences exist like the size of a long, flags to denote if the compiler can cast negative floats to 32bit ints and the sprintf format specifier strings for handling floats and doubles. The data also includes some less expected ones like:



armhf: #define Netdb_host_t		const void * /**/

amd64: #define Netdb_host_t		char * /**/

I'm not at all sure why that is arch-specific - if anyone knows, email codehelp @ d.o - same address if anyone fancies helping out ....

Cross-builds and debclean

When playing with the cross-build, remember to use the cross-build clean support, not just debclean:



dpkg-architecture -aarmel -c fakeroot debian/rules clean

That wasted quite a bit of my time initially with having to blow away the entire tree, unpack it from original apt sources and repatch it. (Once Wheezy is out, may actually investigate getting debclean to support the -a switch).

OK, that's an introduction, I'm planning on pushing the cross-build support code onto github soon-ish and doing some testing of the cross-built perl binaries in a chroot on an armel box. I'll detail that in another blog post when it's available.

Next step is to look at perl 5.16 and then current perl upstream git to see how to get Makefile.SH fixed for the long term.

Introducing pyBit - Buildd Integration Toolkit

nospam@example.com (Neil Williams) — 2012-11-02T20:13:34Z

pyBit - cross-platform package building using AMQP

Message queues provide a simple way to create a distributed, cross-platform buildd toolkit to build packages using a collection of buildds, direct from various VCS clients. pyBit is intended to support rapidly evolving software collections and can support multiple VCS frontends and multiple build backends. Cross building is expected to be supported for some backends. The initial backend uses dpkg for Debian with subversion providing the source and sbuild doing the actual build.

pyBit includes support for cancelling selected builds and using multiple buildd clients per architecture, per platform and per suite.

Hooks are available or in development for subversion and git, other VCS hooks can be added. A RESTful web API provides live build reports and can generate build jobs for specific packages using particular VCS branches on selected architectures to support re-building packages at any point in the development process. Build history is stored using postgresql.

Other buildd systems can rebuild long lists of packages or build lots of binary packages from relatively slow moving source packages. PyBit exists to handle much more rapid software development across a wide range of platforms, VCS inputs and architectures. Buildd clients which are under-used can be tasked with building multiple suites or adding cross-build support. Buildd clients which are over-utilised are easily identified and adding new machines to an existing architecture / platform / suite pool should be easy. Hook activation automatically cancels any ongoing build for the same architecture, platform and suite to avoid wasting time on an interim version.

The emphasis in pyBit is to have fast builds with redundant clients, reliable reporting using a flexible and intuitive frontend. To this end, there is no need for a source package to be uploaded. Depending on the VCS hook in use, builds can happen every time a particular file is changed (e.g. debian/changelog for a native Debian package) or at every push (for a distributed VCS) or whatever is appropriate for a particular software collection. Builds are checked for available build-dependencies and packages re-queued if the build-dependencies are not yet met.

In the longer term, it may well be possible to use more than one server / database combination to support more builds and more platforms.

So far, we've got to a working model and tagged 0.1.0 as the first downloadable release. There is a lot more to do, especially adding more documentation, more VCS hooks, support for more VCS methods on the buildd clients and more buildd client scripts for platforms other than Debian. (The git hook and git source client are expected to let pyBit be self-buildable but a certain amount of configuration will be required for the server and each client which makes it not-quite self-hosting.)

pyBit concentrates on preparing collections of binary packages which can be used to build others, rather than trying to rebuild everything every time - this allows more rapid upstream software development and encourages modular, re-usable software. pyBit will also support rebuilding specific versions, architectures, suites or platforms via the RESTful web API. Access to this frontend can be controlled through any of the standard methods.

Components are loosely coupled via JSON encoded messages sent using rabbitMQ and curl. A new client can be added at any time and it will simply pick up buildd jobs from the relevant queue.

Development has now moved to Github and if anyone wants to look at new clients and new hooks, just contact the team by email or via IRC (#pybit on irc.oftc.net). Current development is based on Debian Squeeze 6.0.6 with backports and also on Debian Wheezy - more testing is welcome. Patches are also very welcome, pyBit is licensed under GPL2+.

There is no intrinsic reason why pyBit could not support any buildd platform capable of taking source from a known location and building a set of binaries. Packages are added to the queues whenever the hook is activated, so adding new packages to the collection is simply a case of triggering the existing hook.

Another interesting challenge for pyBit would be to trigger a hook on a new source code repository and just let it work through every package until everything is built. That probably won't work with the current 0.1.0 code but if that is what you'd like to work on, join the team.

subversion 1.7.5 insane upgrade requirement wasting days and days of effort

nospam@example.com (Neil Williams) — 2012-06-19T21:50:51Z

Upgrading the local version of subversion against a large subversion repository has so far taken three DAYS. It goes through this multi-gigabyte repository in no perceivable order, it goes through every single directory in every single tag and every single branch and refuses to run in any sub-directory. (Even after clearing out some older tags, the tags/ directory is still nearly 8Gb and that doesn't include any binary files.) Each tag within tags/ is 500Mb and contains nearly 6000 sub-directories. Yes, it's large but svn upgrade should still be able to handle it without crippling every machine.

I can't image how svn is going to manage when the server finally gets upgraded. Probably be quicker to dump the entire repo and reimport it.

When I finally give up the will to proceed or simply need to use this LAPTOP for something else, I have to interrupt it with Ctrl-C at which point is starts all over again!

This isn't a slow machine, it's a i5 quad-core T410 with 8Gb of RAM and 1Tb of storage - and svn has made it crawl for days. The only way to pair down the working copy is to delete every tag and branch which means losing data like old build logs and old packages. The repository is this large because it's tracking the development of multiple commercial products which share common code but which also have numerous releases and release updates.

I can't even use svn st on this repo without this completing - I haven't been able to work on this repo since this started. Absolutely insane.

Pondering filing an RC bug on the basis of unjustifiable data loss. We have many machines at work with this repository checked out and if we ever migrate to Wheezy, it's going to mean a WEEK of lost work!!

subversion                    1.7.5-1

So, a warning for anyone else using subversion 1.7.5 with a very large repository: DELETE ALL TAGS and BRANCHES and any other directory anywhere in every single working copy tree on every machine which ever wants to use that copy again before even thinking about upgrading to 1.7.5.

The tags directories don't even need to be upgraded because we only use those to rebuild the code as it was in chroots.

Unspeakably furious about such a completely dumb tool being thrown into the mix DAYS before the Wheezy freeze. Now I have to kill it AGAIN just so that I can suspend the laptop. IDIOTS.

Going to Nicaragua

nospam@example.com (Neil Williams) — 2012-06-09T15:57:00Z

Not DebCamp this year, but I intend to get to more talks than I did in Bosnia and still get some work done on Emdebian Integration into Debian as well as working on as many RC bugs as I can manage during the week.

Possibly easy targets via UDD

A couple of my usual RC bug filter queries:

wheezy-and-sid, ignoring patch, pending, security, claimed and not in main. (483 bugs)

wheezy-or-sid, ignoring merged and done (987 bugs)

Multiarch and debi

nospam@example.com (Neil Williams) — 2012-03-24T11:13:14Z

If you're in the habit of doing sudo debi at the end of a build, it's worth noting a complication with Multiarch.

Sometimes, it's tempting to rebuild (and install) a shared library with a few untested changes, however, if (like me), you have multiarch packages installed, this can cause a surprise:



dpkg: error processing ../libqof2_0.8.4-2_amd64.deb (--unpack):

 trying to overwrite shared '/usr/share/doc/libqof2/changelog.Debian.gz', which is different from other instances of package libqof2:amd64

Bumping the changelog doesn't help:



dpkg: error processing libqof2:amd64 (--install):

 package libqof2:amd64 0.8.4-3 cannot be configured because libqof2:armel is at a different version (0.8.4-2)

dpkg: error processing libqof2:armel (--install):

 package libqof2:armel 0.8.4-2 cannot be configured because libqof2:amd64 is at a different version (0.8.4-3)

The fix for this is, of course, to remove libqof2:armel and all it's reverse dependencies and the packages can't be put back until you've built an armel version of your changes.

The way to backout the test change is a bit longwinded, depending on the complexity of your library:



sudo apt-get --reinstall  install libqof2:amd64=0.8.4-2 libqof-dev:amd64=0.8.4-2 libqof2-backend-qsf:amd64=0.8.4-2 libqof2-backend-sqlite:amd64=0.8.4-2 libqof2-dbg:amd64=0.8.4-2 libqof2:armel=0.8.4-2

Hmm. I think a helper tool could be needed in the medium term here.

This also means that cross-building Emdebian Crush packages from a Wheezy / Wheezy+1 base is again looking as if it will need to happen inside a chroot, albeit with problems for the packages which are dependencies of the build tools (or the cross toolchain) itself, as the Crush packages will inevitably contain modified files. (For example, Emdebian Policy differs from Debian Policy by requiring - not forbidding - compression of debian/copyright.)

Cross-building packages which are intended to be binary compatible with Debian without conversion (including cross-built packages to use on top of Emdebian Grip) shouldn't be too affected. You just need to ensure that where the package exists in two or more versions, all installed architectures are built before any architecture is installed.

This is an additional burden compared to the world of dpkg-cross but it is key to how Multiarch allows cross-building to use sensible dependency resolution.

So, what I think we'll need is an enhancement to debi which can be passed an architecture (possibly an architecture list), debi then looks for both the native arch and the requested architecture(s), fails> if the .changes file for the extra architecture(s) isn't found or proceeds to install all packages for the native arch and arch-dependent packages for the second architecture in the same dpkg operation. As a further enhancement, debi might be able to check dpkg --print-foreign-architectures, check the output of $package:$arch against dpkg -l and either complain or automatically use the Multiarch support to look for the relevant .changes files. debi may even want to check the package (as it's working in the package top directory usually) for Multiarch support before accepting the architecture list option.

Think I need to do some reading of the debi source and see if a patch is workable...

Rubbish in the archive

nospam@example.com (Neil Williams) — 2012-03-04T10:18:27Z

I've been asked (or been criticised through indirect media) about why packages are removed and as I've just done quite a lot of removals at the Cambridge BSP (I've done some uploads too, it's not one-sided), I thought I'd explain my rationale in the hope that it will encourage others to remove more packages and fix those which warrant fixing. This, is an approximation of the kind of scoring I would use when assessing whether to fix a package or remove it. All bug counts are for the entire source package, including all bugs reported against binaries built from that source.

R - Release critical bugs - count the bugs cumulatively - 1 point for 1 bug, 3 points for two (1+2), 6 points for 3 (1+2+3) etc.

U - Uninstallable binary packages - extra points if a -dev package is involved as this will cause FTBFS in reverse dependencies. 1 point for every unrelated package affected. Add extra points if the uninstallable packages are reported as bugs rather than just as notices in the PTS. Add further points for each follow-up message to those bugs. (Filing or following up to a bug report is an indication of user impact as someone had to actually make a contribution.)

B - Blocking other bugs - extra points for every RC bug which is blocking any other bug of any severity.

B - Blocking transitions or release goals - extra points for every bug which is blocking a release transition. (Note this is not the same as being involved in the transition, it matters if the package is blocking the transition by making other packages involved in the transition unable to be rebuilt or migrate.)

I - Interest - popcon is not much use here, "Interest" is a multiplier of the total of previous points. If the package is in testing, multiply the previous points total by the number of days the package has been waiting to enter testing beyond the normal limit. If it is blocking other packages from migrating, add the total number of excess days of all blocked packages. If the package is only in unstable, multiply the previous points total by the number of months the package has been waiting.

S - Security - Count up the number of days the CVE has been waiting longer than a week, multiply that number of days by three and multiply the points so far by the result.

H - Help - whether anyone has done any work at all, including the maintainer. Subtract 1 for any patches, 2 if the patches still apply and work. (It's not much for patches because if a patch exists and works, the excuses not to upload are less believable). Double the points so far for any working patches which have been ignored for more than a month (more than a week for a CVE or packages blocking transitions.

RUBBISH - easy to remember, maybe.

If someone comes up with a UDD query which can resolve the above as an algorithm, a) I'll buy them a beer at DebConf and b) it could be a useful addition to the PTS...

Introducing ladder

nospam@example.com (Neil Williams) — 2012-02-18T09:52:44Z

We needed a new package at work and once I'd written it, I realised that it could well be useful for others, so I did the ITP and the package is now in Debian unstable.

Ladder - Stepwise repository upgrade tool



Ladder creates a local repository of packages required

to upgrade a tarball of a chroot to a new milestone or

software release.

.

The repository can be signed and includes all specified

target packages and dependencies. The repository can then

be distributed and used to upgrade multiple devices in

sequence without needing network access.

The only dependencies are some simple perl modules, reprepro for the repository, apt and gnupg for the signing. None of those need to be particularly recent versions, so once ladder has had a little time in testing, I will be doing a backport to Squeeze. Lenny is a little less likely but eminently possible if someone specifically asks for it. Not that it particularly needs a backport mind, if you want to run ladder on some other Debian-based system, it's simply a case of installing the version in unstable. reprepro was in Lenny at a suitable version for what ladder requires, so any system remotely recent should be able to run ladder without changes.

Normally, whenever you need to upgrade a device, you need to get that device onto a network, get access over the serial port or some form of pre-configured network connection, run the commands and clean up afterwards. OK, now do that again... oh and here are another 40 or 100 devices which all need precisely the same thing done and they all have to be shipped to paying customers today. What does any geek do? Script it.

Ladder is part of that process. There would need to be some scripting / programming support on the devices concerned but ladder makes it easier to put that support in place at the design stage and then automate the actual update without the device needing to get onto a network and, potentially, allowing you to decide how to offer the upgrade (automatic, user involvement, engineers only etc.) Once that is implemented, the device is simply pointed at a locally mounted filesystem which contains the ladder step for the migration. Each step is a SecureApt signed repository which is available at a deterministic location which can be easily scripted and which contains only the packages necessary (with dependencies) to upgrade any device installed at Milestone A to Milestone B. Nice and small, only the stuff you need, whatever architecture you like and if you need to migrate from Milestone A to Milestone D, then you create a few steps on the ladder and automate the migration from A to B to C to D. All you need is enough space on whatever local media you need to use to plug into the devices. Then copy the media enough times, start the interface on each device and do something useful whilst the upgrade happens. Then, when the next lot need to be upgraded, you already have the media... Oh and ladder does not need to run on the same architecture as the devices - all package handling is entirely architecture-neutral. So create the ladder steps and media on your fast x86 machines and then let the slower SSD devices take their own sweet time via automation. Simple - maybe, hopefully.

I'm expressly talking about devices here because this is where the original requirement arose - embedded devices which don't have direct / automatic / accessible network ports or even a working network configuration preset. These aren't new images to flash onto the device either, these are apt repositories, so only the stuff which needs to be changed is changed which saves a lot of time writing to slow SSD storage.

If the mechanism sounds familiar, yes, it's using the same principles as xapt, multistrap and emdebian-grip but I thought it could be useful, so I generalised the interface (a bit) and uploaded.

Let me know via the BTS if you need tweaks to the support. Ladder isn't particularly about getting desktop machines from Debian Potato to Debian Wheezy because you need to identify a target package to be the top point of the dependency chain which gets into the step repository and all devices should start with the same packages at the same versions. Normally, the kind of devices I use at work have a single top level application which is the sole user interface and which directly or indirectly depends on all the other software required for that device. Whilst the target package value could take a few carefully selected packages instead of just one and there is support for an extra packages field, it's not intended / expected to be useful for desktops. This is for use with multiple devices which all have the same package selection, are at the same milestone at the start and which all need to migrate smoothly to whatever milestone is intended - one step at a time. Ladder could be used to migrate between Debian releases and I've included example config files to support that, but the principle usage is with internal / proprietary repositories based on a common Debian stable platform where the user interface software is managed via identifiable milestones and where users have no direct control over installing packages. It's a production / manufacturing support tool more than a user / admin support tool. If you want to manage servers and desktops which allow users to install (or request to be installed) arbitrary packages, there are plenty of tools which already support this and are used by DSA in Debian for exactly these tasks. Ladder is not puppet but Linp didn't sound like a good name...

Ladder step repositories only include the binary packages, not sources - so if the migration is not going to be done in-house and you're expecting to distribute these steps to users somehow, ensure that the original source repository is available online for the Debian / free software packages concerned. If you can't do that (because the source is proprietary), you possibly shouldn't be distributing the binaries as a user update mechanism in the first place because each step will include packages from the core Debian system which need to have the sources available when distributed.

Inevitably, the ladder source package already contains two POT files for translations of the runtime messages and the POD based manpage. I expect to need to expand / clarify the manpage in due course, so don't rush to translate it yet as it's likely to change.