Codehelp blog

The script isn't fully automated yet, I'm running it manually and watching for errors, but I do now have a simple script (~100 lines of perl) which runs edos-debcheck against each of the supported Emdebian architectures for unstable-grip and picks out those packages which are missing from the subset of packages which Emdebian monitors for Grip.

I was hoping to make these scripts into a normal Debian package but there are some limitations in that the scripts need to assume various pieces of Debian infrastructure. e.g. I started off using rsync to pull in the Packages files (along with Release and Sources) because apt has annoying behaviours with multiple architectures. (Emdebian Grip processes 7 architectures simultaneously on blavet.debian.org). Peter Palfrader kindly arranged for a local mirror to be available and now the entire Packages processing can be done using symlinks. Much, much better.

Actual package movement data comes directly from dak via projectb - something else which makes packaging these scripts for use on non debian.org boxes a bit hard.

Once again, I am indebted to the EDOS team (please, please keep working on edos-debcheck - it hasn't had uploads recently) because I tried to get this working with germinate but failed. (Colin - I'll be looking to borrow some of your time to work out what was going wrong.)

The dependency resolution script now means that I can meet release team requirements that when foo is updated in sid to depend on libbar2 but Emdebian Grip only has libbar1, the scripts will automatically pick up libbar2 and add it to Emdebian Grip. The current sid version gets pulled in and (the theory goes) will migrate into testing-grip as an almost automatically valid candidate. Processing of the dependency can happen within hours of the updated version of foo arriving in Emdebian Grip but as long as it happens within a few days I think everyone will be fine with it.

I've got a sync script too which can periodically run across the entire suite and check for packages which have slipped the net. I'll have to see how often that needs to run but once a week doesn't sound too painful.

Processing for Emdebian Grip is very fast. Most of the time is taken uploading:

2012-01-23 22:24:42 add sid-grip deb main armhf libfltk-images1.3 1.3.0-5em1

2012-01-23 22:24:52 add sid-grip deb main i386 libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:07 add sid-grip deb main mips libfltk-images1.3 1.3.0-5em1

2012-01-23 22:25:25 add sid-grip deb main powerpc libfltk-images1.3 1.3.0-5em1

2012-01-23 22:26:00 add sid-grip deb main amd64 libstdc++6-4.5-dev 4.5.3-12em1

2012-01-23 22:26:34 add sid-grip deb main armel libstdc++6-4.5-dev 4.5.3-12em1

(It was a lot longer, comparatively, when I was having to rsync and then dget instead of just read from the local filesystem.)

I've also been working on an init script which will gradually work through the subset of packages by order of Priority - approximately what a new architecture (like armhf) would do. Note that Emdebian Grip supports armhf already. The only slight concern is whether the init script needs to be throttled back to not flood the queues with hundreds of uploads an hour (which it could end up doing, at least for the first 24 hours).

Logs of package movements are available: http://www.emdebian.org/grip/logs.php

Packages can be searched too: http://www.emdebian.org/grip/search.php

More on Emdebian Integration on the Debian wiki: http://wiki.debian.org/EmdebianIntegration

Now all I need is a fix for #655919 FTP team?

A continuation of my intermittent series on Emdebian and prompted by a query from Paul Wise, I thought I'd cover some of the uses to which a device running Emdebian can be put. It is a bit long this one...

So, yes, there are general purpose computers which could run Emdebian but there are common features amongst general purpose computers:

1. Lots of different tasks must be supported - lots of different packages installed, each user could quite easily use a distinct set of packages from another user.




2. Multi-user support - even if any one machine is only used by one person, a general purpose computer must have support for adding another user - not just a system user, a real user with a login, shell, possibly bookmarks and browser history etc.




3. Connectivity - general purpose computers (and that includes mobile phones) must connect to a variety of other computers to be at all useful.




4. Multi-modal data input - there needs to be support for a mouse and a keyboard or touchscreen and just as likely, software support for accessibility interfaces like an on screen keyboard and switch inputs.




5. Storage for user data - the system admin can't know how much data users are going to add and if it is general purpose, it's probably going to support media players, image viewers, even office software. The data files managed and written by such software tend to be large and tend to accumulate into large collections. This means that a general purpose machine cannot anticipate the full storage requirements, so has to allow plenty of room for user data storage. If there's so much storage space available, why use an OS which is primarily designed to take up less space? Just use the full version, Debian.

Now compare with the common features of special purpose or single-purpose computers:

1. Single task only - a splash screen (without support for removal) covers all the boot information, the single task starts as a daemon and if the user quits the task, the task initiates a shutdown, not quit. Every effort is made to prevent the task from exposing the underlying OS, including auto-restarts if it does crash.




2. Single-user support - indeed, user is not the typical user. user in this case is a non-login user without a shell, created by the postinst of the task package purely to support running the task without root privileges (because if the task doesn't need privileges, it shouldn't retain privileges).




3. Single mode input - don't expect these machines to support 105 key keyboards. You'll be lucky to have 5 keys which are really just buttons. The button won't just enter 'J' into the machine, it will fire off a command to charge your bank or start a drill or shutdown a malfunctioning service. (Each button will often be multi-functional as well, depending on context). Some devices will replace the mouse with a touchscreen but many are fitted in places where touchscreens simply won't survive, where keys have to be able to withstand being submerged in water (or oil) for hours. Many machines simply won't have mouse support at all. Many will not have any typical external connectors. Indeed, units which go into dirty or wet environments certainly won't have standard external connectors of any kind. Think boats and factories. Internal battery packs with custom hot-fit replacements, buttons encased in 4mm of clear plastic, screens viewable only through protective mesh / filters.




4. Restricted connectivity - if the task doesn't need networking, no networking hardware is fitted. If the task doesn't need USB, no USB hardware is fitted. If the task doesn't need serial connections, there will probably be an internal serial connector but also a Warranty void if removed sticker. Don't expect an RS232 inside the case either, this will be a flat cable connector which you'll need to attach to a breakout board which correctly connects the cable wires to the RS232 port pins. That mapping is not standardised (why would it? only service engineers are approved to use it.)




5. Constrained user data - the task is in complete control and determines what the user can view, store and access. Storage can be expensive and a power drain, so if the task doesn't need Gb of storage, don't fit the machine with Gb of storage. Profit.

So what are these mysterious single-use computers running Emdebian? It's hard to tell because the task has no particular reason to tell the user anything about the OS. Perhaps a different way to phrase the same question would be: Why use a general purpose OS for a special purpose computer?.

• Single task does not mean single process - doing one thing can involve doing a couple of different things behind the scenes, so a multi-tasking kernel and userspace is extremely useful in providing a responsive machine without wasting money on unnecessarily powerful hardware. Multithreading support is also valuable.




• Software reuse makes good business sense - why write a single-usage operating system when the device needs a multi-tasking OS anyway and suitable ones already exist?




• Minimal software - no point packaging Xorg or GNOME when the user interface is a tiny QVGA framebuffer and the device only has 128Mb of RAM. By all means get the core system (cairo, pango, Qt, gettext, dpkg) from Emdebian but the actual code which performs the task of the device will generally be bespoke and, yes, that generally means proprietary, closed source, utterly non-free. This is where GNU/Linux wins over iOS and Windows - the graphical software is trivially separated from the core software and tested as such. If your device has only monochrome text output (and plenty of those still exist), what possible reason is there to use a graphical toolkit? Use the multi-tasking support of the OS to handle inter-process communication and understand the licensing - job done. Again, these devices have deliberately limited storage capacity, the smaller the OS, the lower the cost of the device, within certain constraints.




• Debugging compatibility - this is the big one. It's hard debugging the task software on device with the differences in architecture and connectivity. Why not use the same packages on the same OS on the development machines as on the devices? All the power and all the screen resources to properly investigate bugs, simple way to provide an emulator for translation support and other activities (like user interface training). There will always be bugs which have to be debugged on device - timing issues, hardware issues - but why not make at least a proportion of the likely bugs easier to fix by using binary compatible libraries on both the big desktop machines and the small devices?

This is why Emdebian Grip is popular for these machines - because Emdebian Grip is binary compatible with the equivalent Debian suite and when a bug appears in the high level user interface, it is much easier to debug that on the desktop than on device. Debian stable is a fantastic OS for commercial development - the stability of the OS makes detection of interface bugs much easier. It is rare that developers have to investigate whether bugs in their UI come down to a bug in the underlying OS. That saves time.

The machines themselves? I'm sure others can come up with other examples but these are some which come to my mind (no particular order):

• Control interfaces - solar power controllers, robot controllers - anywhere that a low power board can monitor responses from other machines and do something intelligent about reported errors. Temperature limit responses, pressure limit responses, encoding all of those in assembly or VHDL is hard, so use a system which provides for more creative result handling.




• Electric car charging points - very simple user interface, single task which doesn't complicate things by expecting the user to tap on something to start the main application - just process the card details, charge the customer and provide power.




• Similarlyelectronic point of sale devices - EPOS. The staff on the shop floor don't want to be waiting for an X server to start up and then using a menu. The device starts up and runs one task (checking barcodes, printing labels, counting stock), if the task is shutdown, the machine shuts down.




• Communication aids - my area, a machine which presents a simple UI on non-standard framebuffer hardware, lots of audio output but little other connectivity. Resilience is very important, so solid state is vital but why provide more solid state storage than the device can actually needs?

Anywhere where the device simply needs to DoTheRightThing in a variety of unpredictable circumstances, you're going to need a general purpose OS to gather the data and produce the right result. Anywhere where the device has only a single task, you're going to want to not provide access to the rest of the system. There's no point providing "general purpose access" when the device doesn't have "general purpose hardware". There's no point fitting general purpose hardware if the device does not use it. That is a waste of money, a waste of resources and causes unnecessary delays if one component or other becomes unobtainable or changes behaviour. Limit your exposure to hardware bugs and get the product out to the user more quickly and more reliably, everyone is happy. Fit non-standard hardware and you can fit custom hardware which better matches the profile of the device usage - why use standard speakers capable of CD quality across a vast frequency range when the device only makes very very loud alert noises which can be done at lower sample rates and with narrow range, high amplitude, speakers.

There are all sorts of estimates for the number of computers now in the average Western home. It's worth noting that the vast majority of those computers are not PCs or laptops or even game consoles and routers - all instances of general purpose machines with general purpose access / connectivity. A lot of the computers in your home are embedded in machines like white goods, media controllers (set top boxes and the like). Some of these will still use low level firmware written entirely in assembly or COBOL. More complex ones already have a general purpose OS inside yet constrain that OS with a simple, user-friendly interface which is tailored to the hardware actually fitted. It might be cool to connect to your set top box over Bluetooth but really, is that actually going to sell more units? Only if there are other high-end services written for the device which the average customer won't want to see pushing up the price of the base unit. Some machines (particularly in automotive uses) will need to be using Real Time operating systems and Safety Critical operating systems but even some of those are based on general purpose systems - just old versions which have been thoroughly tested.

Finally, consider that for a lot of these devices, the customer is not you. The customer is another business who then put the device into something else which is installed into a factory production line which produces something which goes into creating a consumer product you find in Tesco or on Amazon or bundled in with another service like your TV/broadband contract. There is absolutely no reason for these devices to provide general purpose computing to you, even if the device itself is part of something else which can provide such services.

OK, this one is meant for Planet Debian...

One of many, many changes in the latest lintian is vendor profiles and, thanks to a heads-up by Niels Thykier, Emdebian will have working profile support in the next upload of emdebian-grip. (The only reason it's not already in Debian is my own fault for not uploading when I thought I had the time to upload.)

$ lintian --profile emdebian-grip drivel_3.0.2-1em1_amd64.deb 

$ lintian --profile debian drivel_3.0.2-1em1_amd64.deb 

E: drivel: debian-changelog-file-missing

E: drivel: copyright-file-compressed

W: drivel: copyright-without-copyright-notice

E: drivel: description-contains-invalid-control-statement

W: drivel: binary-without-manpage usr/bin/drivel

So the em1 version implements Emdebian Policy for Emdebian Grip. Clean for Emdebian Grip, just as the Debian package is clean prior to the changes.

I expect this to dramatically improve the processing of Emdebian packages, both for Grip and for the cross-built flavour, Crush, once that actually starts up.

Thanks to the lintian team for this support. Now if there was some way of backporting this version of lintian to Squeeze, I could also use this at work to suppress really annoying lintian warnings about non-standard suite names. (The whole point of using a non-standard suite name is to keep our stuff separate from the normal Debian/Emdebian stuff for licensing reasons etc.) Update: of course, I didn't check the PTS for lintian before writing that, so hence didn't notice that the backport already exists! Thanks again to Niels for the prompt. I've now got another package to create for work. Update2: Thanks to Joerg Jaspert for the tip that all lintian versions get backported directly from unstable as an exception on ~bpo. The work package is ready, so this is going to make things a lot easier when building stuff on stable.

In other news, the same version of emdebian-grip will include support for integrating Emdebian Grip into Debian itself. This too will use vendor-specific support, this time an internal vendor name which just needs to work on the "buildd".

(It's not quite a buildd, Emdebian Grip doesn't build anything, it's all post-processing. It's just that the processing of Debian packages for Emdebian Grip will look a bit like having a second buildd working on the packages uploaded by the existing buildd. The process itself is still developing...)

language        translated     fuzzy     untranslated

-----------------------------------------------------

  da               103           4             4

  fr                54          17            40

  pt                54          17            40

  vi                54          17            40

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Set the changelog message to version 2.1.15+t0 first, closing any relevant bugs!

`dch -v 2.1.15+t0`

neil@sylvester:tdeb-test$ dch -v 2.1.15+t0 TDeb test

neil@sylvester:tdeb-test$ dpkg-gentdeb -t

Building TDeb update: 2.1.15+t0

 (192 entries)

Discard doc/pod/1/da/multistrap (65 of 177 strings; only 36.72% translated; need 50%).

Discard doc/pod/1/da/device-table.pl (5 of 19 strings; only 26.31% translated; need 50%).

Processing untranslated files for pod/multistrap (1) . . .

Processing untranslated files for device-table.pl (1) . . .

Processing de translations for pod/multistrap (1). . .

Processing de translations for device-table.pl (1). . .

Processing fr translations for pod/multistrap (1). . .

Processing fr translations for device-table.pl (1). . .

Processing pt translations for pod/multistrap (1). . .

Processing pt translations for device-table.pl (1). . .

dpkg-gencontrol -pmultistrap-tdeb -Pdebian/multistrap-tdeb -cdebian/control

dpkg --build debian/multistrap-tdeb ../multistrap-tdeb_2.1.15+t0_all.tdeb 

dpkg-deb: building package `multistrap-tdeb' in `../multistrap-tdeb_2.1.15+t0_all.tdeb'.

neil@sylvester:tdeb-test$ dpkg -c ../multistrap-tdeb_2.1.15+t0_all.tdeb

drwxr-xr-x root/root         0 2011-06-08 21:06 ./

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/

-rw-r--r-- root/root      6427 2011-06-08 21:06 ./usr/share/locale/pt/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/

-rw-r--r-- root/root      6963 2011-06-08 21:06 ./usr/share/locale/vi/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/

-rw-r--r-- root/root      6789 2011-06-08 21:06 ./usr/share/locale/fr/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/

-rw-r--r-- root/root     16311 2011-06-08 21:06 ./usr/share/locale/da/LC_MESSAGES/multistrap.mo

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/pt/man1/

-rw-r--r-- root/root     33963 2011-06-08 21:06 ./usr/share/man/pt/man1/multistrap.1

-rw-r--r-- root/root      4471 2011-06-08 21:06 ./usr/share/man/pt/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/vi/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/fr/man1/

-rw-r--r-- root/root     33969 2011-06-08 21:06 ./usr/share/man/fr/man1/multistrap.1

-rw-r--r-- root/root      4708 2011-06-08 21:06 ./usr/share/man/fr/man1/device-table.pl.1

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/

drwxr-xr-x root/root         0 2011-06-08 21:06 ./usr/share/man/da/man1/