Codehelp blog

OK, been quiet for a few days, at least on the blog, but in accordance with typical behaviour (for me at least) a quiet blog means a lot of things going on elsewhere. The smallest was a fix for deb-gview - combining a fix for opening files in the current working directory with a fix for the imminent dpkg 1.14.17 which modified the .changes file format a bit. That was relatively easy and I've uploaded 0.2.1 to unstable. gvfs hasn't made testing yet so there's no point in waiting for deb-gview to migrate before 0.2.1.
The largest change? Depends how you define large:

1. Large number of changes to package files - emdebian-tools 1.0.0
2. Large effect on Emdebian workload - dpkg 1.14.17

dpkg

dpkg bugs closed - emdebian-tools 1.0.0 pending - dpkg-shlibdeps is now able to look into directories containing libraries used by cross-built binaries provided that the right environment variable are set. Closes: #453267. dpkg-buildpackage will set PKG_CONFIG_LIBDIR (but not override an existing value) in case of cross-compilation so that pkgconfig finds .pc files in the directory specific to the target architecture. Closes: #439979.
These two changes will have a major effect on the overall workload required within Emdebian and give us a sane cross-building environment in Lenny which is ABSOLUTELY £$^£@&*! FANTASTIC. Currently, 1.14.17 is in experimental but I've installed it on my system for testing purposes so that I can complete emdebian-tools 1.0.0:

emdebian-tools 1.0.0

Now here, I'm making some progress. A bug has been bothering me for some time with how emdebian-tools needs a Debian primary mirror to calculate various dependency information and other apt cache stuff whilst taking into account the host architecture. Simon and I now think that we can do the dependency stuff with a continually updated dpkg-cross repository and I can use libcache-apt-perl for the rest. So whereas dpkg-cross might still exist into Lenny, it is looking like apt-cross might not - at least not as an application. However, I'm not taking that step with emdebian-tools just yet. In the meantime, I'm simply using debconf to replace the automated selection of a Debian Primary Mirror and passing that mirror to apt-cross - something I should probably have setup a while ago.
As well as that bug fix, I've added a check to emchain so that users are warned if they don't actually need to run it, added the mirror support to the various scripts and removed the workarounds now fixed by dpkg.
'dpkg-buildpackage -a' is usable for nearly all packages once you have dpkg-dev >= 1.14.7. emdebuild >= 1.0.0 is only necessary IF:

1. You want to modify/update Emdebian patches (not just apply them) - i.e. if you have commit access to Emdebian SVN.


2. You need the one remaining wrapper: 'X-Build-Cross-Libtool: yes' which is my next main target. This is a package-specific problem and each incidence needs a package-specific solution via an appropriate patch. As such, the wrapper is likely to stay in place in emdebuild for some considerable time - at least until someone can rebuild the entire archive with a test that can identify which packages are still affected.


3. You want the build checks currently implemented by emdebuild but which I plan to spin out during the life of emdebian-tools 1.x. I want to standardise those build checks and abstract them into possibly a lintian-compatible check set.

Also, there should now be no need for any wrapper scripts that anyone may have devised to mangle LD_LIBRARY_PATH or PKG_CONFIG_LIBDIR - indeed,
explicit support for these within dpkg means that it is to be considered harmful from now on to touch these variables, except possibly in some specific corner cases.

(Note: this is a long post)

Package: deb-gview

Use of GnomeVFS: "Individual package files or packages referenced in a changes file can be viewed from local or remote filesystems using GNOME VFS."

Main changes: src/dvarchive.c but also src/main.c.

Port Status: Complete and tested.

Prerequisites

Do not run any code before completing this step

You must ensure that gvfs, gvfs-bin and gvfs-backends are all installed and operational before running any of your modified code. Thereby hangs another tale. The application itself probably needs to depend on both gvfs and gvfs-backends. gvfs-bin provides test binaries like gvfs-info.

You need to ensure that the following command operates correctly:

$ gvfs-info http://www.google.com

display name: /

edit name: /

type: unknown

size: 0

attributes:

  standard::display-name: /

  standard::edit-name: /

  standard::content-type: text/html

  id::filesystem: type='http',uri='http://www.google.com',mount_prefix='/'

Note that this is run as the normal user, not under sudo and not as root. - that just complicates things. If you find that this works with sudo but not without, read the bug report against gvfsd.

If you get this output:

$ gvfs-info http://www.google.com

Error getting info: The specified location is not supported

read the bug report, find and kill the borked gvfsd process and ensure that gvfs-backends is installed before attempting any other gvfs* commands (including via your modified source code).

Starting

1. Online copy of the GIO docs. online copies of the GVfs docs. I mainly use Anjuta where both sets of the docs are available via libglib2.0-doc >= 2.16.1 and devhelp or via your localhost using dwww.


2. #include <gio/gio.h> to replace #include <libgnomevfs/gnome-vfs.h>.
   (May sound obvious now but examples are not that frequent in the GIO docs as part of libglib2.0-0.)


3. The GIO docs do give some info on porting variables:
   
   
   • GnomeVFSHandle * vfshandle; became GFile * ghandle; GInputStream * stream; in deb-gview
   
   
   • Extra variables needed for deb-gview: GVfs * vfs; GError * result; GFile * gfile; GFileInfo * ginfo;. GError itself is not new but it can catch you out in the port because so many GIO/GVfs functions can use it and you find yourself using g_clear_error(&error) a lot or GLib will complain about trying to replace data in a GError.
   
   



4. Separate path and URI handling - GIO and GVfs have separate filename/path handling for paths on a local filesystem and paths on remote filesystems and the two cannot be mixed. This was a big source of bugs during the port (it may be a UTF-8 issue) and in the end I went for a gboolean flag in the context that this particular viewer was handling a file from a remote filesystem (e.g. http://localhost/incoming/deb-gview_0.2.0_amd64.changes, http://ftp.debian.org/debian/pool/main/e/emdebian-tools/emdebian-rootfs_0.9.1_all.deb or http://incoming.debian.org/foo_version.changes) and not from a local path (e.g. /opt/debian/deb-gview/deb-gview_0.2.0_amd64.changes or ../../debian/deb-gview/deb-gview_0.2.0_amd64.deb). Personally, I found g_file_is_native to be unsuitable for this purpose and as I had a "user-entered" path available, I simply use if ((g_str_has_prefix (path, "..")) || (g_str_has_prefix (path, "/"))) but YMMV, depending on how you obtain the path itself.

   
   

   Keep track of the type of input file when handling GFile - g_file_get_path will fail if passed a GFile that needs g_file_get_uri and vice-versa. These failures output messages to the console so they are relatively easy to track but not only will the call fail but the GFile itself is likely to get corrupted, leading to other bugs and messages to the console. g_vfs_get_file_for_path(vfs, path) and g_vfs_get_file_for_uri(vfs, path) (which obtain the GFile from GVfs in the first place) also fail if passed the wrong type of path.

   
   

   In some situations, g_file_get_parse_name can be used to get a "normal" char* representation of the path to the GFile but I was only able to use this in specific circumstances and only when the path and URI handling flag was implemented.

   
   

   This all complicates the port because gnome_vfs_make_uri_from_input did not have this separation issue.




5. Checking if a file exists - if (!g_file_query_exists (gfile, NULL)) may seem easy but unless issues with the URI and path are handled cleanly, this simple check will fail even though the file exists. The problem arises because GFile appears to mangle a URI with a path function and vice-versa. The GFile ends up with an invalid path and hence there is no valid file to check so the query fails - but without indicating that it's actually an internal error in GFile rather than a valid path that simply does not exist.




6. Reading files in one go - vfsresult = gnome_vfs_read_entire_file (vfstext, &changes_size, &contents); gets replaced with g_file_load_contents (cfile, NULL, &contents, &changes_size, NULL, &result); (note the order swap between the size and the buffer to use). Once read, the buffer can be handled as before. This is one of the easier changes in the port, once the above issues are all cleared up. On the other hand, GIO and GVfs do appear to handle relative paths more easily than GnomeVFS.




7. Checking VFS is active -

   
   

   vfs = g_vfs_get_default ();
   
      if (g_vfs_is_active(vfs)) {




8. Reading files as a stream - result = gnome_vfs_read (vfshandle, readbuffer, skip, &bytesread); becomes bytesread = g_input_stream_read (stream, readbuffer, skip, NULL, &result);. Fairly straightforward as long as you use:

   
   

   if (result) {
   
   g_warning("Error: %s", result->message);
   
   g_clear_error (&result);
   
   }

   (or something equivalent to get the error message to the user), every time you use &result.

   
   

   vfs_uri = gnome_vfs_uri_new (gnome_vfs_make_uri_from_shell_arg (filename)); becomes filestream = g_file_read (deb->ghandle, NULL, &result); (there's that &result again).




9. Checking the stream remains open.

   
   

   stream = (GInputStream*)filestream;
   
          if (g_input_stream_is_closed (stream))
   
          return FALSE;




10. Seeking inside the stream - YMMV but seek never appeared to work well for me in VFS so I had to read into a dump instead.

    
    

    gpointer dump; 	        
    
            dump = g_new0(gchar, seek); 	        
    
            result = gnome_vfs_read (deb->vfshandle, dump, seek, &bytesread); 	 
    
            g_free (dump);

    
    

    In GIO, seek is much easier: g_input_stream_skip (stream, seek, NULL, &result); - just handle that &result again.




11. Closing the stream - gnome_vfs_close (vfshandle); becomes g_input_stream_close (stream, NULL, &result);.




12. Getting information on files - if ((vfsinfo->type != GNOME_VFS_FILE_TYPE_REGULAR) || (vfsinfo->size == 0)) becomes

    
    

    ginfo = g_file_query_info (ghandle, G_FILE_ATTRIBUTE_STANDARD_SIZE, G_FILE_QUERY_INFO_NONE, NULL, &result);
    
     if (0 == g_file_info_get_attribute_uint64 (ginfo, G_FILE_ATTRIBUTE_STANDARD_SIZE))
    
    {
    
           dv_show_error (DV_ERR_NO_FILE, parent);
    
           return;
    
    }

    
    

    (dv_show_error is a user-facing error report func) and, yes, there is a g_clear_error(&result); later.

Summary

This took me quite a bit of time and the bug in gvfsd was a significant hindrance so ensure that you test with gvfs-info before thinking there is a bug in your code. (Also remember that if you are reading files from http://incoming.debian.org/ then those files will disappear once each day when the cron job moves them into the pool. ) I recommend developing and testing against http://localhost/ during the port.

The GIO docs are relatively thin (especially for GVfs) and that is not just because GIO is still new. Starting a new application using GIO and having a working GIO/GVfs setup in place already would be much easier so new applications will work. As ever, the problems really show up when trying to port an application that has already worked around the various GnomeVFS bugs and now has to both remove those workarounds and implement one or two other changes for GIO.

Ensure your application has Depends: gvfs-backends - until the bug in gvfsd is fixed, it would be unwise to put gvfs-backends in Recommends: because that will almost inevitably result in any user without Recommends: enabled passing a call to GVfs at least once without an available backend, resulting in a broken gvfsd process and a completely broken application until the user either kills the process or logs out and logs back in. Even identifying that issue in a bug report is not straightforward - hopefully the bug report and this post will help. (Yes, depending on how you configure the ported application, it is entirely possible for this broken gvfsd process to break all file operations in your application, even local ones.) Unfortunately, there is no call (currently) to take down GVfs - you can only test to see if it is already active and that check is not sufficiently intelligent to detect the broken gvfsd process.

Subjectively, GIO does appear smoother than GnomeVFS and maybe faster. Now I need to get on and try to port other applications to GIO.....

Update

After contacting Sebastian Dröge, I've found that the problem with GVfs isn't in the deb-gview code, it is a permission problem within GVfs - the current deb-gview CVS code works with remote URIs when run under sudo which isn't ideal but at least indicates the nature of the problem. Once I have confirmation and/or gvfs is fixed, I'll skip deb-gview 0.1.6 and upload 0.2.0.

** (deb-gview:31002): WARNING **: file-read:http://ftp.debian.org/debian/pool/main/g/gpe-tetris/gpe-tetris_0.6.4-1_i386.deb:The specified location is not supported

Hmm. This is with current CVS of deb-gview, already substantially modified to work with GIO and GVfs as libgnomevfs is deprecated. I wanted to look at the port because quicklist and gpe-filemanager will need similar ports.

$ dpkg -l 'gvfs*'
ii gvfs 0.2.1-1 userspace virtual filesystem - server
ii gvfs-backends 0.2.1-1 userspace virtual filesystem - backends

I have an RC bug in deb-gview that I need to fix and I wanted to do the two jobs in one but it's looking now as if I'll need two uploads, 0.1.6 with a trivial fix to unstable just to fix the existing bug and 0.2.0 to experimental with a lot more changes but partially broken functionality. It's a shame because I've got everything else working using GIO - it is only the GVfs part that is giving errors.

Anyone else using GIO and GVfs with http:// in Sid?

OK, the misery of the HP laptop continues.
In hibernated mode, picked up the laptop, open the lid, tap the power button and the lights come on then the lights go off. Followed by the fan and the screen. Try the power button again - nothing.
Power is still getting into the machine - the power indicator works.
Power does not appear to be getting any further. Bugger.
(That's a nice present on my birthday isn't it!)
Only option now is to remove the hard drive and take the disasterous road labelled "Customer Service" as it's under warranty. (That way I can avoid having to revoke my laptop key - it's not my main key but it's got a lot of signatures that I'd rather not loose.)
Damn and double damn. Waste of money, feeling miserable and $%^@& annoyed. I'm almost tempted not to bother getting the useless thing repaired.
So, I'm back on my iBook laptop which will probably now have to go back to running unstable.

This could be fun - gpe-filemanager is not currently available for Emdebian because it depends on libgnomevfs which in turn brings in a variety of other dependencies ending up with liborbit2 which simply does not (and appears cannot) cross build.
Thankfully, GNOME 2.22 deprecates libgnomevfs in favour of GIO which is already part of libglib2.0-0 and only needs other stuff via plugins.

I need now to port deb-gview to GIO and, in the process, probably see if gpe-filemanager can also be ported to GIO. After all, a GUI without a file manager is going to look a little odd.

I'm still working on libqof2 upstream too so it looks like I won't be doing much on Lenny for a while - not directly at least.

I expect to have deb-gview 2.0.0 depending on GIO and in Lenny and maybe quicklist too. (Typical, no sooner does quicklist migrate from Gtk1 than it also needs to migrate from libgnomevfs.) Then there's all my usual Debian and Emdebian stuff.

Unsurprisingly, there is a deal of common ground between a SQLite backend and a GDA backend. Both need simple routines that convert library data into SQL statements but each has very different ways of reading that data back from storage.
I've spent today doing upstream work (whilst also getting my central heating system replaced). Gutted the house, gutted the library - what's wrong with that?
However, I've decided to make these new functions private to the library rather than exporting them. Simple reason really - they are one-way routines.
One day, QOF will support UPDATE statements and one day a bit later it will also support DELETE and even DROP but those require changes within the library like "unregistering" objects that the current code simply doesn't expect.
Eventually, this kind of SQL support can make a usable query interface for quicklist and a few other stalled projects (like DWI). The principle gain is to allow QOF objects to evolve and adapt but retaining compatibility will be a challenge. (QuickList is likely to only support a read-only mode for some objects and require that others are renamed before being amended.)
The other advantage of this refactoring is that the SQLite backend in libqof2 will be a lot smaller than before. Migrating the code hasn't increased the size of libqof2 either because so much deprecated code has been removed when bumping the SONAME.
I'm sticking with the roadmap of not looking to release libqof2 until after Lenny but that gives me plenty of time for internal reworking to get the GDA backend functional.
So far, I have been able to create a test GDA data source, add a table and data, delete the data and drop the data source. Not bad! (I test with SQLite but XML, MySQL, PostgreSQL and other ODBC connections are supported by GDA without changes to the QOF code.)
As ever, implementing new functionality needs a new test case so that has also been started - only 20 tests so far but worth doing.
Other changes before libqof2 - at the request of the original developer - will include updating QOF-Generator to work with libqof2, updating the old HTML example pages to provide a "QuickStart Guide" and other introductory documentation. Whilst I'm doing that, I'll probably also work on the Emdebian website documentation that also needs improved coordination.
Also note that pilot-qof and gpe-expenses are already libqof2 compatible. All that will be needed is some way for those to provide the username and password variables that some GDA connections would need. (The QOF GDA code already has support for such settings, it just needs to be implemented in the respective frontends.)
First set of changes are now in QOF CVS.
(Yes, CVS - I just don't see the appeal of all those new-fangled RCS versions - subversion is about as "modern" as I need. I just don't need the hassle that comes from git fanboys (or any of the other RCS versions) - nor do I need the hassle of migrating CVS to SVN.)
What I would say is that of the various SVN WWW interfaces, websvn stinks, view svn is a bit better but the only one I really like is Trac. For a single large project, Trac is much more friendly IMHO.

Bubulle asked about whether anyone has computed the number of bugs open (and by inference close) and I've just found a link from the DD statistics pages:
Debian Bug Tracking System News - These are the bugs touched during the last four hours. The data will be updated in 40 minutes.
Active Bugs 110443 (+9) 9230042 kb
Archived Bugs 366389 (+0) 24156850 kb
Debian NEW closable bugs 41
Source packages in incoming.debian.org 19
Package movement for testing +17 -2
Orphaned packages 360
Popularity contest Debian 76404 (+0) Ubuntu 448702 (+0)
Total registered dpkg/apt Users 525106

There's an RSS feed too.

 noapic nolapic noirqfixup