Codehelp blog

I've got a new, fast, server and I've migrated my homepage to Drupal6. This has given me a chance to experiment with i18n updates.

I've got a section for translations and I can write blog entries for this section using drivel. The new i18n blog is a separate feed, not syndicated on planet, intended for anyone who wants to know about the translation status of any of my packages - whether upstream or Debian. POT files will be uploaded for each package and each message (should have) links to the VCS where the current PO files can be found.

I could post continual messages to the debian-i18n mailing list but this way I can put specific content (like messages about which parts of the project I'm changing) as well as updates on which translations are lagging etc. I hope it'll be easier for translators to find the relevant data instead of having to search through the debian-i18n mailing list archive.

I'm not sure about comment handling within the i18n section at this stage. I think I'd prefer to get input via normal email and bug reports.

This, more general, i18n blog category will remain on my main blog and syndicated with planet. I'll use this for posts that aren't specific to one particular project.

gnome-doc-utils has a fairly idiosyncratic layout - help/package.pot, help/lang/lang.po - which doesn't suit things like podebconf-report-po that I want to use to call for translations and updates.

This little snippet creates the relevant directory from a gnome-doc-utils Makefile.am:

report-po: Makefile
if [ ! -d po ]; then \
mkdir po/ ; \
fi
cp $(PACKAGE)-help.pot po
cd po ; \
for l in $(DOC_LINGUAS); do \
if [ ! -L $$l.po ]; then \
ln -s ../$$l/$$l.po . ; \
fi; \
echo -n $$l: ; \
msgfmt -c --statistics $$l.po; \
rm -f messages.mo; \
done


Also, I thought gnome-doc-utils updated the PO files automatically but I couldn't get it to do it, so I've got this snippet too:


check: update-po

update-po: Makefile
for l in $(DOC_LINGUAS); do \
xml2po -u $(top_srcdir)/help/$$l/$$l.po $(top_srcdir)/help/C/$(PACKAGE).xml; \
done


po4a, including the new po4a-build script, has been released and po4a (0.37.0-1) has been uploaded to Debian. This release also includes a fully updated Spanish translation, including the new po4a-build documentation, thanks to Omar Campagne.

po4a-build will be used as a build-dependency by emdebian-rootfs and svn-buildpackage (both due to have updated releases in a couple of days). (po4a-build itself was developed and tested within these packages - as 'genmanpages'.)

po4a-build is intended to make it as easy to produce translated documentation as it can be to produce the current untranslated content.

When po4a prepares the translated content as POD or DocBook XML, the final documentation can then be built using po4a-build. Both the untranslated and translated content is built as a single process, updating the POT files at the same time.

Existing build instructions are replaced by a single call to po4a-build and a simple configuration file is used to tell po4a-build how to build each element and which binary packages will include the translated and untranslated content.

See po4a-build (1) and po4a-build.conf (5) from the installed po4a package.

Feel free to try it out with any package that builds translated documentation, *especially* if it also builds binary gettext translations of program output without using autotools, e.g. perl scripts using $(MAKE) but not ./configure.

This should be a useful step in encouraging more packages to translate their script output and manpages. It is trivial to add translation to packages using the full autotools|intltool setup, now po4a-build brings the same level of ease to packages only using $(MAKE). (If someone wants to investigate how CMake can do the same, let me know.)

http://alioth.debian.org/projects/po4a/

http://alioth.debian.org/frs/download.php/3183/po4a-v0.37.0.tar.gz

http://freshmeat.net/projects/po4a/releases
(0.37.0 announcement pending approval by FM)

po4a creates the meta files needed to generate the manpages but the actual generation process is a little awkward and tends to become heavily customised.

For emdebian-rootfs, I've now received the Portuguese translation and I've been improving the genmanpages script. It's now more general and easier to configure for different source packages. There remain a few hidden assumptions which I hope to fix in future.

#!/bin/sh
set -e

# Copyright (C) 2006-2009 Neil Williams
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see .

# name and location of the config file
# relative to the top level source directory of the translations
# i.e. the directory containing ./po/
CONFIG="doc/po4a.config"
# POTFILE path
POTFILE="po/emdebian-rootfs.pot"
# the binary packages that will contain generated manpages
BINARIES="emdebian-rootfs multistrap"
# the binary packages with manpages in Section 3.
HASMAN3="emdebian-rootfs"
# the binary packages using DocBook XML & xsltproc
XMLPACKAGES="emdebian-rootfs"
# the DocBook XML files (TODO: let this take paths)
XMLFILE="emdebian-rootfs.1.xml"
# the XSL file to use for Docbook XSL
XSLFILE="http://docbook.sourceforge.net/release/xsl/current/manpages/docbook.xsl"
# the POD files (TODO: let this take paths)
PODFILE="em_multistrap"
# the binary packages using POD
PODPACKAGES="multistrap"

# remember to add something to the clean:: rule in debian/rules
# to remove each binary package sub-directory.
# $(RM) -r doc/emdebian-rootfs doc/multistrap

# below this point, no changes should be needed.

wrap_langs () {
if [ -z "$LANGS" ]; then
echo "$1 $2" >> $CONFIG
else
echo "$1 $2 $3" >> $CONFIG
fi
}

if [ -f po4a.config ]; then
cd ../
fi
if [ -f ../po4a.config ]; then
cd ../../
fi

# calculate the langs, automatically.
LANGS=`ls po/*.po 2>/dev/null || true`
if [ ! -z "$LANGS" ]; then
LANGS=`ls po/*.po | sed -e 's/.*\/\(.*\)\.po/\1 /' || true`
fi
if [ ! -z "$LANGS" ]; then
# LANGS should not include the specification, just the root.
LANGS=`echo $LANGS | sed -e 's/_.*//'`
LANGS=`echo $LANGS | sed -e 's/@.*//'`
LANGS=`echo $LANGS | sed -e 's/\+.*//'`
echo "[po4a_langs] $LANGS" > $CONFIG
else
echo > $CONFIG
fi
wrap_langs "[po4a_paths]" "$POTFILE" "\$lang:po/\$lang.po"

for d in $BINARIES; do
for l in $LANGS; do
mkdir -p doc/$d/man/$l/man1/
if [ "$d" = "$HASMAN3" ]; then
mkdir -p doc/$d/man/$l/man3/
fi
done
mkdir -p doc/$d/man/man1/
if [ "$d" = "$HASMAN3" ]; then
mkdir -p doc/$d/man/man3/
fi
mkdir -p doc/pod/$l/
done
mkdir -p doc/html/

for file in `ls doc/xml/*.xml`; do
xmllint --format "$file" --output "$file"
LOCAL=`basename $file`
wrap_langs "[type:docbook]" "$file" "\$lang:doc/xml/\$lang/$LOCAL"
done
# oops, this needs to be configurable
for file in em_multistrap; do
wrap_langs "[type: pod]" "$file" "\$lang:doc/pod/\$lang/$file"
done

# use -k to create all XML even if untranslated or the XSL breaks the build.
po4a -k 0 $CONFIG

for d in $BINARIES; do
for P in $XMLPACKAGES; do
for X in $XMLFILE; do
XML_CATALOG_FILES="/etc/xml/catalog" \
xsltproc -o doc/man/ --nonet $XSLFILE doc/xml/$X
done
if [ "$d" = "$HASMAN3" ]; then
mv doc/man/*.3 doc/$P/man/man3/
fi
mv doc/man/*.1 doc/$P/man/man1/
done
for POD in $PODPACKAGES; do
pod2man $PODFILE > doc/$POD/man/man1/$PODFILE.1
done
for l in $LANGS; do
for P in $XMLPACKAGES; do
for X in $XMLFILE; do
XML_CATALOG_FILES="/etc/xml/catalog" \
xsltproc -o doc/man/ --nonet $XSLFILE doc/xml/$l/$X
done
if [ "$d" = "$HASMAN3" ]; then
mv doc/man/*.3 doc/$P/man/$l/man3/
fi
mv doc/man/*.1 doc/$P/man/$l/man1/
done
for POD in $PODPACKAGES; do
pod2man doc/pod/$l/$PODFILE > doc/$POD/man/$l/man1/$PODFILE.1
done
done
done
for l in $LANGS; do
rm -rf doc/xml/$l
done
rm -rf doc/pod/

# the rest is specific to emdebian-rootfs - ignore.

# HTML versions, not packaged.
xsltproc -o doc/html/ --nonet \
http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl \
doc/xml/emdebian-rootfs.1.xml

for i in `ls doc/html/*.html`; do
iconv -t utf-8 -f iso8859-1 $i > doc/html/tmp
sed < doc/html/tmp > $i -e 's:charset=ISO-8859-1:charset=UTF-8:'
done
rm -f doc/html/tmp


OK, more to be done here but it has a basic idea. The generated files are separated out into directories determined by the binary package name into which the manpages will be installed.

This gives a .install line something like:
doc/emdebian-rootfs/man/* ./usr/share/man/

This is important because it reduces the number of changes needed to add a new translation - in effect, only the $lang.po file itself.

It should at least give some ideas of what can be done.

A bit of an unusual situation with drivel - I'm taking over maintenance from Neil McGovern and I've updated various parts of the package during that process. However, to avoid swamping Neil in wishlist [l10n] bugs during the time taken for the BTS etc. to switch over, I'm not sending a call for updated translations at this time, even though most of the current translations are out of date.

drivel is also quiet upstream and although there is an unreleased version in SVN that I will probably package once 2.0.3-4 is in Squeeze, I've checked the upstream po/ and none of the existing translations have been changed in SVN, beyond predictable changes in the line numbers where strings appear. There are a small number of new translations but most of these are also out of date:

$ msgfmt -c --statistics po-svn/ar.po
43 translated messages, 219 fuzzy translations, 43 untranslated messages.
neil@holly:test$ msgfmt -c --statistics po-svn/dz.po
305 translated messages.
neil@holly:test$ msgfmt -c --statistics po-svn/oc.po
91 translated messages, 214 untranslated messages.
neil@holly:test$ msgfmt -c --statistics po-svn/th.po
212 translated messages, 93 untranslated messages.
neil@holly:test$ msgfmt -c --statistics po-svn/zh_HK.po
297 translated messages, 4 untranslated messages.

Bizarrely, one has even been removed:

$ msgfmt -c --statistics po-current/no.po
149 translated messages, 29 fuzzy translations, 96 untranslated messages.

I'm pondering whether to join / take over upstream for drivel, at which point it becomes trivial to make a useful call for translations, get all the PO files updated both in the current package and in SVN and then make a fully updated release.

drivel already includes bg.po ca.po cs.po de.po el.po en_CA.po en_GB.po es.po fi.po fr.po ja.po lt.po nb.po ne.po nl.po no.po pa.po pl.po pt.po pt_BR.po ru.po rw.po sq.po sr.po sr@Latn.po sv.po vi.po zh_CN.po zh_TW.po and all of those are out of date.

So don't file i18n bugs against drivel just yet, wait for the call. Thanks.

A bit of news on langupdate - things are moving along and it is mostly working, including a new no-op mode when installed on Debian so that it doesn't try to replace files from packages that do not use TDebs yet.

I've added support for /etc/locale.gen so that any configured locale is automatically picked up and selected for TDeb support. In addition, extra languages can be specified via the LANGUAGE environment variable. When languages are de-configured (or the LANGUAGE variable is unset or not given), unwanted TDebs can be automatically purged using the -r command to langupdate. As with previous releases, all cache data is downloaded to a dedicated directory under /var which is cleaned out after each run, including the downloaded TDebs themselves.

The one bug that is holding back a release is that although langupdate can identify TDebs for binary packages that have the same name as the source package, I'm having problems retrieving the name of the source package from the libapt-pkg (C, not perl) library without resorting to mass fork/exec for apt-cache showsrc. I'm not sure if it is a repository problem or an apt problem.

langupdate is now sufficiently advanced that I've asked for translations of the 41 gettext messages included with the binary. So far, I've got three translations in place and 10 days more before the translation deadline expires.

If anyone fancies helping out with the C/C++ code for langupdate, the
SVN is available via Emdebian.


# langupdate -?
Usage:
langupdate [OPTION...] - updates language support for emdebian

Apt frontend to match the Emdebian TDeb support with the installed
packages and supported locales.

Help Options:
-?, --help Show help options

Application Options:
-n, --dry-run Only show the cache data, do not install
-m, --mirror=mirror_name Use the specified TDeb mirror instead of the Emdebian default.
-r, --autoremove Remove TDebs for unsupported locales.
-v, --verbose=integer Specify verbose level [0-4]
--suite=codename Specify the apt suite name [default unstable]
--version Show version information and exit.

Emdebian TDebs provide a method for splitting all translation files out of
packages and into separate TDeb packages, organised by the locale root and
source package. This allows individual users to only install translations
files for the locales supported on their own machine and only for the
packages installed at the time.

Note that this functionality is not available for Debian TDebs.


po4a

- or PO for anything. PO is the de facto standard for translations and po4a tries to help maintainers prepare packages for translation.

1. I18n (internationalisation) is making a package accessible for translators and is an upstream/maintainer action. i18n is done once for all translation support and updated when the content changes (typically during the build).


2. l10n (localisation) is using i18n support to create an individual translation for a specific locale. l10n is done many times per package, with individual translations being updated each time the i18n content changes. (Tdebs will support this area but are a separate topic from po4a.) PO files that exist when the POT is updated are also updated, tagging modified strings as "fuzzy".

The natural burden, therefore, falls within l10n but without i18n, l10n is all but impossible (it means having to do the i18n "outside" the package).
I'm using po4a and I've been doing various translation-support stuff now for a few years, so I'm going to add po4a into my blog entries as a series to help maintainers make their own packages i18n aware.
This is particularly important for:

1. all Debian native packages, particularly scripted packages. My example will be based on perlpod documentation often embedded wthin the perl scripts themselves as well as DocBook XML for manpage generation. po4a supports many other formats.


2. All manpages (or other documentation) created by Debian maintainers

To make things easier, I'll be creating a new category in my blog ("i18n") for such entries. This is helpful for me too because the first time I used po4a (current versions of emdebian-tools) I basically got things wrong and made more work for myself than was necessary. This entry primarily concerns providing i18n support from manpage sources and perl scripts. i18n support in compiled packages is a separate issue. The fixes described here will be included in the next version of emdebian-tools (which will be uploaded to Emdebian but not into Debian until after Lenny).

Terminology

• PO - (Portable Object) files contain the POSIX C locale strings abstracted from the "master document" which in many cases is the source file in the relevant package(s), alongside the translated version of that string (if any). Multiple master documents are typically combined into a single collection of translatable strings. PO files are translated into individual locales, one PO file per locale.


• POT - PO Template - the definitive collection of translatable strings from which all PO files are prepared and updated. The POT file is never translated directly, it is maintained, updated and packaged by the maintainer.


• gettext - the binary application that handles the translatable strings. gettext abstracts the strings out of the source files during the package build and inserts the translated strings back into the text actually displayed to the user at runtime. If no suitable PO file exists, the user gets the untranslated string, exactly as it appears in the POT file. If a suitable PO file does exist but the strings have been modified since translation, gettext drops "fuzzy" translations that do not match precisely and the user gets a "mixed" translation containing some translated strings and some untranslated. Most translators try to maintain a 100% translation because the ugliness of the "mixed" output is a partial incentive to keep up with the updates.

Using po4a

po4a needs a config file where the maintainer keeps all metadata relating to the master documents and the relevant formats. po4a then converts the master documents into a collection of strings for translation and puts those strings into a POT file (a PO template file). The maintainer packages the POT file (and optionally sends the POT file to relevant mailing lists seeking translations to be returned as PO files) and keeps the POT file up to date during the build process. With a config file (e.g. called po4a.config), all the maintainer needs to do is call:

po4a po4a.config

po4a then parses the list of files defined in the config file, generates the lists of strings, collates the lists and identifies where the same string is used in multiple files and writes out a POT file containing references to where the strings are used and the unique strings.

po4a config file

Comments start with '#' in the usual way. Instructions are enclosed in square brackets.
po4a_paths defines where the POT file will be put (it can be extended later but I'm keeping this simple). The name of the POT file is important for compiled packages but can just be the name of the source package for scripted packages and/or manpages.

[po4a_paths] po/emdebian-tools.pot

type identifies *each* master document to be processed by po4a. The syntax is well suited to being generated from some other list within the package but I'll give just a snippet of the final output here:

[type: xml] doc/xml/em_autobuild.1.xml

[type: xml] doc/xml/embug.1.xml

[type: pod] emtargetcmp

[type: pod] emdebcheck

That's it. The complexity of my current script was entirely unnecessary.

Also, po4a is more than capable of handling multiple document types within the same source package and the same POT file which is something that I had missed so far. One more hack that can be removed.

All I do now is use xmllint to "clean-up" the DocBook XML and pod2html and some XSLT to generate HTML documentation for the website. The loops to do those tasks already contain the list of files to be processed by po4a, so a few echo commands is all that is needed to create po4a.config at build time and run po4a to generate a combined POT file for all strings in the package.

#!/bin/sh

CONFIG="po4a.config"

echo "[po4a_paths] po/emdebian-tools.pot" > $CONFIG

for file in `ls doc/xml/*.xml`; do

	xmllint --format "$file" --output "$file"

	echo "[type:xml] $file" >> $CONFIG

done

for file in emdebcheck emtargetcmp emprunecross Emdebian/Tools.pm \

 em_installtdeb emrecent emcache; do

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2html -outfile html/EmdebianTools.html -title $file < ../$file

	else

		pod2html -outfile html/$file.html -title $file < ../$file

	fi

	if [ "$file" = "Emdebian/Tools.pm" ]; then

		pod2man ../$file > man/Emdebian::Tools.3

	else

		pod2man ../$file > man/$file.1

	fi

	echo "[type: pod] $file" >> $CONFIG

done

po4a $CONFIG