Gunnar Wolf

Last week (July 7-13) was basically hell on Earth, for me and for the group that somehow got the name Cabras locas, of which I am part since I joined the National Pedagogical University, where I worked full-time 2003-2005.
It was, yes, the first of my officially three weeks of Summer holiday at IIEc-UNAM, so no problems here. So, why hell on Earth? Because we were in charge basically of anything related with information flow, retrieval and manipulation at the 11th International Congress on Mathematical Education, in Monterrey.
What we thought would basically be one or two days of hard work followed by six days of relaxed vacations (we had even planned to have an internal seminar, showing off the shiny stuff each of us is working on) became... A mind-boggling eight day experience where we worked over 12 hours a day on being human replacements for Google, SQL engines, full-text parsers, report generators, printer watchdogs, and in general lines, just a bunch of unhappy firemen, ready to be called off for whatever task was necessary.
We did have, of course, several calm periods every now and then. We even had to learn how to look busy while doing something compeltely unrelated (that would explain, for example, a couple of low-hanging bugs I fixed for Debian, or some dozens of lines of code I could get off my head).
But my advice for whoever reads this: Don't trust people with long database-handling experience. Specially when they insist that hand-capturing a thousand registers is preferrable (i.e. less error-prone) than parsing three separate databases and discarding duplicates. And, of course, specially when this person is your boss, which is enough of an argument to have it his way.

I think I should follow up on Victor's lament. Yes, we have a Rails application which works fine most of the time... But quite often, throws out a segmentation fault I just have been unable to pin-point. It might be related to rmagick, the only non-pure-Ruby component I am using (and I'm tempted to try minimagick instead, even if I prefer in-memory operations than on-disk, piping an image and slurping it again).
Victor came up with an easy script to check the server - but to reduce the impact it has (I was running a single Mongrel instance, which meant, whenever it dies the whole system becomes inaccessible for everybody; I replaced it with a mongrel_cluster of five processes, plus pound as a easy-to-use balancer which looks quite nice), the very simplistic and to-the-point script did no longer work.
Anyway... Ruby rocks ;-) I'm sharing this with you mostly because I am sure some readers will find more than one useful construct, not because it is precisely beautiful code. And besides, we should work on fixing the cause, not the consequence, of the bug! :)

#!/usr/bin/ruby
require 'yaml'
confdir = '/etc/mongrel-cluster/sites-enabled'
restart_cmd = '/etc/init.d/mongrel-cluster restart'
needs_restart = false
 
(Dir.open(confdir).entries - ['.', '..']).each do |site|
  conf = YAML.load_file "#{confdir}/#{site}"
  pid_location = [conf['cwd'], conf['pid_file']].join('/').gsub(/\.pid$/, '*.pid')
  pid_files = Dir.glob(pid_location)
 
  pid_files.each do |pidf|
    pid = File.read(pidf)
    begin
      Process.getpgid(pid.to_i)
    rescue Errno::ESRCH
      warn "Process #{pid} (cluster #{site}) is dead!"
      File.unlink pidf
      needs_restart = true
    end
  end
end
 
system(restart_cmd) if needs_restart

I usually don't like me too comments... But this is something that really disappoints me of my otherwise-favorite development framework. I must echo Matt Palmer's comment on Luke Kanies' entry:
Ruby. Has. A. Distribution. Problem.
Nice, good read. Sadly, many Rails pushers see distributability as something very minor, something that should not worry Rails developers right now, as there is too much other serious work to be done - Better UTF8, a clearer language, better performance... And besides, any programmer can live well with gems. (yes, that's all taken from a rant I had with a very convinced person)
My gripe is that... Rails is no longer a small, fringe project. Rails is an enterprise-grade development framework, with thousands of deployed production systems. And if they don't start to act responsably, if the Rails developers keep pushing said problems as low-priority, the Rails developers' (that is, their users) culture will become rigid - and will constitute a serious harm to Rails' future.
Distributability and packageability is not only for OS distributors. Not only we Debian zealots care about software being easily packageable. By using Ruby Gems, you dramatically increase entropy and harm your systems' security.
Read Luke's text for more details. It is quite worth the time.

This is the third plugin I have been working on for Rails - It's the one that still needs most work to be fully usable (for what I originally envisioned it), but is almost there - I have not touched it (nor the other two I recently presented here, acts_as_catalog and Real FK because real-life has demanded my time on various other fronts.
Anyway, this is IMHO the most interesting of the three plugins I've written so far.
So, following what I did with the other plugins: You can go to acts_as_magic_model's main page, or jump straight to its SVN repo. And the basic description follows:
Rails plugin to allow for extensible models, where inter-model relations are inferred from the database's structure, and not necessarily from explicit declarations in the models.
This means, when one of your models acts_as_magic_model, this plugin will try to infer all the basic relations (has_many, belongs_to and has_and_belongs_to_many) its base table has with other tables in the database, adapting the model correspondingly.

Using the plugin

In order to use this plugin, you only have to declare it in your model. As an example, say you have the proverbial project-management system, where you have projects (each of which can be of several different types) and people, and a HABTM relation between people and projects. So, you have the following tables (expressed as migrations here):

    create_table :people, :force => true do |t|
      t.column :name, :string
    end
 
    create_table :project_types, :force => true do |t|
      t.column :name, :string
    end
 
    create_table :projects, :force => true do |t|
      t.column :name, :string
      t.column :project_type_id, :integer
    end
 
    create_table :people_projects, :force => true, :id => false do |t|
      t.column :person_id, :integer
      t.column :project_id, :integer
    end

    class Person < ActiveRecord::Base
      acts_as_magic_model
    end
 
    class Project < ActiveRecord::Base
      acts_as_magic_model
    end
 
    class ProjectType < ActiveRecord::Base
      acts_as_magic_model
    end

As stated yesterday, I have recently worked on some plugins for Ruby on Rails, and I do think they are usable (and useful!) for audiences larger than myself. So, today I'll present here the second such plugin.
acts_as_catalog implements a really simple but really common pattern - Catalog-tables (or should I say catalog-models?) Granted, the amount of code this plugin saves is not very large, but it does add maintainability for our code-base, and includes basic unit tests. Of course, you can also taka a look at its SVN repo.

Models

A catalog is defined as a table with only a name column of string type, and with a unique index on it (this means, does not allow for duplicate values). Of course, more attributes can be added to a model which acts_as_catalog - they will just be ignored by this plugin. This plugin allows you to specify your model definition as:

  def Mytable < ActiveRecord::Base
    acts_as_catalog
    belongs_to :some_other_table
  end

  collection = Mytable.collection_by_id
 
  another_col = Mytable.collection_by_name

Migrations

You can specify all the catalogs you need to create for a specific migration with a single instruction from inside your self.up method, by giving a list of catalog table names to create_catalogs:

  def self.up
    create_catalogs :countries, :states
    ...
  end

  def self.down
    ...
    drop_catalogs :states, :countries
  end

Tests

This plugin provides the base functionality to include in your unit tests ensuring the catalog is properly declared. Just include ‘catalog_test_helper’ and include the CatalogTest mixin in your test classes, declare the model name, and off you go. This is, a complete unit test for you Mytable catalog model could be:

  require File.dirname(__FILE__) + '../test_helper'
  require 'catalog_test_helper'
 
  class MytableTest < Test::Unit::TestCase
    include CatalogTestHelper
    def setup
      @model = Mytable
    end
  end

I have mostly shifted my main development language to Ruby, as most of the work I do tends to be pretty well covered with Rails. That's not new, however - What is new is that I feel I have got to the point where I actually have something to share. And it feels very good.
So my next couple of posts (three, actually) will be covering some plugins I've written. They are, of course, just sketched - but they work reliably enough to be deemed as useful by their author ;-)
Today I will start by blogging about RealFK. Of course, I will be copying from its documentation/description - I wrote it as well, so I'm not really plagiarizing my own site. I'm not suing self. You can of course also browse the RealFK SVN tree.

References

Rails is built upon a "dumb database" concept. Rails assumes the database will not force restrictions upon the user - i.e. that the model validators are all there is. But I disagree ;-) I was raised with PostgreSQL, not with MySQL, and I do believe the database can and should impose sane values - It should impose trust on the data..
So, if you want to create a relation field, properly acting as a foreign key and refering to another table, instead of doing it via the ActiveRecord::ConnectionAdapters::TableDefinition#column (or even ActiveRecord::ConnectionAdapters::TableDefinition#references) methods, declare them with add_refrence:

  def self.up
    create_table :proposals do |t|
      (...)
    end
    create_table :proposal_types do |t|
      (...)
    end
    create_table :proposal_statuses do |t|
      (...)
    end
    add_reference :proposals, :proposal_types
    add_reference :proposals, :proposal_statuses, :null =>false, :default => 1
  end

  def self.down
    remove_reference :proposals, :proposal_types
    remove_reference :proposals, :proposal_statuses
    drop_table :proposal_statuses
    drop_table :proposal_types
    drop_table :proposals
  end

Join (HABTM) tables

When you define a simple has_and_belongs_to_many (HABTM) relation in your model, you are expected to create an extra table representing this relation. This is a very simple table (i.e. it has only a row for each of the related table‘s IDs), and it carries foreign key constraints.
Please note that join tables are not supposed to carry any extra columns - If you need to add information to join tables, think twice and better use a has_many :elements, :through => :othermodel declaration, and create the othermodel table manually. To create a HABTM table representing that each person can attend many conferences and each conference can be attended by many people:

  def self.up
    create_table :people do |t|
      (...)
    end
    create_table :conferences do |t|
      (...)
    end
    create_habtm :people, :conferences
  end
  def self.down
    drop_habtm :people, :conferences
    drop_table :people
    drop_table :conferences
  end

Zack, Wouter: Yup, Perl 6's scheme changes to reduce confusion to the programmer and to make the language better Huffman coded. Why?
As for the programmer, it comes out of observation - Changing the sigil depending on what you are expecting to get back (i.e. a hash has a '%' sigil, but if you only want one value off it, you use the '$' sigil, as it indicates a scalar variable) confuses more often than it helps - Specially newcomers. Besides, it will be easier for the interpreter itself - The sigil will become an invariant but required part of the variable name itself. Hence, Martin will be happier, as there will no longer be different namespaces under which to store your variables. Oh, BTW, Martin: Not only Perl behaves this way. Ruby also uses that concept - There, you have local variables (no sigil), instance variables ('@'), class variables (ugly '@@') and global variables ('$'). I remember in old (Quick|GW)BASIC you had numeric variables (no sigil) and string variables (postfix sigil '$') as well. Any others?
Now, about the Huffman coding part, WTF?
It is one of the design goals of Perl 6 to make the most often used operators be the shortest, so you, programmer, don't wear out your keyboard and lose your valuable fractions of seconds typing i.e. '->' for a method call when you can just type a '.' - Similarly, currently you have to type too many characters for the (too common) dereferencing - If you have a reference to a hash of arrays of arrays of objects on which you want to call methods, you have to do $something->{key}[0][0]->boo - Whereas with Perl 6 you will only have to do $something{key}[0][0].boo
Anyway, I cannot elaborate much further, as I've been -at best- just a bystander. But yes, large changes loom in Perl's future. Let see what comes on...
And when. Sigh.

What's the best way to join a community, any community? Little by little.
I have been working with Ruby on Rails for well over a year already, even given a talk (at Encuentro Nacional Linux y Software Libre ENLi) inviting other people to use this very nice and well thought out environment. But, so far, I've been only a end-user, not really giving anything back besides minor bug reports.
Well, it's not as much as to say that I'm now a contributor - this is just a first statement of intent. I have worked a bit on several bits of code I keep repeating and hand-copying over my different Rails projects. Of course, that's completely not DRY - And completely not nice. So I decided to stop advancing on several projects, and learn my way on stting those bits of code as Rails plugins.
So far, it's been quite simple - and a good excercise on proper separation. I started, of course, with the easiest bit of code I could think of as useful in a general sense, and packaged it up as acts_as_catalog (of course, proper SVN tree and very basic, introductory README. And, of course, as I keep progressing with this work, I'll keep adding some plugins to my currently quite empty RubyForge page.
Anyway... Little by little, time will come where we have to think more seriously on how to properly bring together the Ruby on Rails style to work more tightly with Debian. Currently, the two have such different points of view on how to manage and ship components that I am not sure we will be able to truly bridge them together... But it is definitively worth trying. Hmh, looks like a task for Debconf! ;-)

I just spent the productive part of the last couple of days going over several alternatives, as I didn't want to do the most obvious thing.
But I ended up doing it.
I think I did it carefully... And in a restricted system.
Still, having a Web-facing script that executes a password-changing script running with Sudo-granted privileges... No matter how much correctness and sanity checking it involves...
Makes me feel dirty.

Cannot help but laugh and share.
I've been triaging and trying to reproduce some oldish bugs on pkg-perl's packages. Some bugs are no longer there, some have to be forwarded upstream, and so on. Usual tasks, yes.
Until I stumbled with #406227. I just have to laugh and share! Hope nobody feels ashamed - The bug is the result of different people coding maybe under pressure and with quite different mindsets :)
For some reason I fail to understand, the submitter's test case (rot13 implemented over a HTTP proxy) is invoked in the report as ./rot13 2>/dev/null. Of course, when trying to debug a bug report, the first thing to do is not to ignore STDERR. So, off goes the 2>/dev/null. What happens next?

0 gwolf@mosca[2]/tmp$ perl ./rot13 &
[1] 4394
0 gwolf@mosca[3]/tmp$  GET -p <a href="http://localhost:8080/" title="http://localhost:8080/">http://localhost:8080/</a> <a href="http://www.debian.org/<br />
Can't" title="http://www.debian.org/<br />
Can't">http://www.debian.org/<br />
Can't</a> locate object method "filter" via package "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)" (perhaps you forgot to load "UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)"?) at /usr/share/perl5/HTTP/Proxy/FilterStack.pm line 126.
500 EOF when chunk header expected

WTF... Well, at least the program name gives me a clue... Lets try to "decrypt" the error message...

  gwolf@mosca[4]/tmp$ echo 'UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160)' | rot13 
HTTP::Proxy::BodyFilter::simple=HASH(0x604160)

hrm... How comes the filter is filtering its own code and only then refusing to find itself!? Ok, time to open up the manpage - Remember, I'm only group-maintaining this pacakge, I am not yet at all familiar with it! Ok, so the core of the filter is when the submitter states:

my $proxy = new HTTP::Proxy();
$proxy->push_filter(response => new HTTP::Proxy::BodyFilter::simple(sub { tr/a-zA-z/n-za-mN-ZA-M/; }));

While the manpage states it should be invoked as:

my $filter = HTTP::Proxy::BodyFilter::simple->new( sub { ${ $_[1] } =~ s/foo/bar/g; } );
$proxy->push_filter( response => $filter );

Of course, once looking at it, the answer is simple: The submitter left out which element to act on in the anonymous function body - The ${ $_[1] } =~ part. Adding it makes gur svygre jbex nf rkcrpgrq... Err, sorry - makes the filter work as expected.

Now, bonus points: For the non-Perlers out ther: How come we get the namespace translated as well? Oh, that's very simple: In Perl, as in Python (and concievably other languages I'm unaware of), the object is passed to any of its methods as the first argument. Functions in Perl receive their arguments via @_ (read: the default array). And, of course, the tr (regex-based transliteration) takes by default the first thing it sees - the object itself. And what happens when you apply a (string-oriented) regex to an object? Of course, it gets stringified - which, by default, in Perl means converting it to the closest possible description: "a hash reference blessed as an object of the class such-and-such at this memory location". That string gets worked on, and we get UGGC::Cebkl::ObqlSvygre::fvzcyr=UNFU(0k604160). This proxy does not die on very-very-short web pages, where the whole content fits on one iteration of the code (although it does not work correctly - the text remains unaltered, of course, as it was not worked on), but if the request spans several chunks, the second time the filter is called, it will be... just gibberish.

Oh, and what about the extra ${ (...) } around $_[1]? Oh, simple: The string is passed as a scalar reference, so it can be modified in place. Yes, it's the Perl way of pass-by-reference instead of pass-by-value (the default behaviour): Of course the parameter is only passed as a value. Only that the value is incidentally a reference - but who cares? ;-)

Anyway... Many oddities. I would implement the module in a completely different way, and it looks quite backwardish in my book. But then again, TIMTOWTDI.

Ok, the time has come. I have postponed this change too much - But finally, after three months of having hired my Dreamhost account (and stating so in this very blog), I finally made the switch from Jaws to Drupal.
Now I only hope I don't flood any planets with my RSS (I was careful to check the dates are consistent, but you never know), finish moving over my static content (I carried over about half of it already), play a bit with the theme, and... that's it! :D
Anyway, I promised my oh-so-not-generic-but-what-the-heck Jaws (0.7 at least) to Drupal (5.x) migration script. It worked like a charm - Ok, I only used it to move blog and photo albums/entries, but that's at least the most typical use AFAICT.
Now, I'll have to understand still some more terms and details in Drupal. For example, WTF? Why was jaws_to_drupal.pl renamed to jaws_to_drupal.pl.txt? (stated content-type, I guess) Why do my uploaded tar.gz files get renamed to tar_.gz even if I explicitly requested to allow .tar.gz suffix uploads? (same thing, I guess)

Some days ago, my RSS reader found Mark Jason Dominus' Help.pm - Yes, the module is (so far, at least - I could not find it on CPAN) only published as a blog post. But don't let that fool you - It's a beautiful (and simple!) Perl module that can help developers that are too lazy to go look up methods in the man pages.

Perl's introspection capabilities are not behind other dynamic languages' (i.e. Python's or Ruby's, speaking only about what I'm familiar with). However, it's used much more seldom, partly because Perl does not ship by default with an interactive console (such as Ruby's irb or Python's regular behaviour when called without an input script). Of course, writing a Perl console is an easy task, and good Perl consoles exist, although its use is not part of the Perl culture.

But of course, just glancing over MJD's code made me come up with a simple, yet useful, way to use introspection in Perl, usable as a simple one-liner. Say you want to look at all of the methods provided by IO::File:

gwolf@mosca[25]/tmp$ perl -e 'use IO::File; print join(", ", grep {defined &{"IO::File::$_"}} sort keys %{"IO::File::"}), "\n"'
binmode, carp, confess, croak, gensym, new, new_tmpfile, open, qualify, qualify_to_ref, ungensym

gwolf@mosca[26]/tmp$ perl -e 'use IO::File; print join(", ", grep {defined ${"IO::File::$_"}} sort keys %{"IO::File::"}), "\n"'
VERSION

Of course, I'll be uploading Help.pm to Debian very soon - And, why not, I think I'll add a way for it to query on different symbols, not just on methods. And the simple binary to call from the command line. Sounds very much worth it ;-) Thanks, MJD!

Ok, so two people replied to yesterday's post about triple negations - Dato (by email) and MadCoder. Both, said basically the same thing: || false and && true are silly noops. And yes, knowing this, I added them. Why? Clarity... At least having them at the end of a test shows the statement is of conditional nature (and not just another obscure attempt to do ${DEITY}-knows-what). They at least look cleaner than a one-line-squashed if block in a makefile. To me, at least ;-)

But... If you noticed this post's title, it goes beyond this comment - One of the most benefical effects I noticed when I installed Jaws 0.7 (over 0.6, of course) is that I no longer had the swarms of spambots flooding me - I often had hundreds of comments a day, and nowadays I hardly get any spam. Now, I fail to see what is so strange in my blog's comment forms (it does not even have any obvious Javascript, although it does obfuscate a bit the source of the captcha image). And you are not the first Debian people to complain you cannot post comments to my site. Strangely, few non-Debian people have ever complained.

And yes, the spam has stopped, almost completely.

So, Debian guys: Are you human?

I'm packaging Ruby's PDF::Writer module for Debian - It is a simple module and (almost) ready for upload. But anyway, it carries some issues I had to bring to debian-legal's attention, mainly, five files licensed under Creative Commons licenses (specially two of them, which are under its NonCommercial variant - clearly non-free), so I'm repackaging the .orig.tar.gz into a +dfsg version.

But I know I'm a lazy and sometimes stupid bum. Even more, the package will be group-maintained by the pkg-ruby-extras team, so we must be as careful as possible not to forget to remove the non-free material - To remove two scripts, and to replace three images with free equivalents I just made. But hey, tell me if this does not feel ugly to you. At least to the bits of you who learnt human grammar:

clean::
	# Make sure we strip out non-DFSG demo files from the orig.tar.gz
	[ ! -f demo/qr-library.rb -a ! -f demo/qr-language.rb ] || false 
	[ $$(md5sum images/bluesmoke.jpg | cut -f 1 -d ' ') == 0586eca5af7523ab871609eceb44724a ] || false
	[ $$(md5sum images/chunkybacon.jpg | cut -f 1 -d ' ') == a000b1917142ce332fd3474f0722cd6f ] || false
	[ $$(md5sum images/chunkybacon.png | cut -f 1 -d ' ') == 927feec1cbbf23c4d89a4a5ad88e6d0f ] || false

Triple negations. How nice.

As of late, I've shifted quite dramatically my sysadmining/development activities. On the development front, although Perl is still my mother tongue and I maintain many systems I wrote with it (and my main involvement in Debian is through the pkg-perl group, of course), I've been largely switching over to Ruby - both under the Rails framework and doing standalone stuff.

But somehthing that's new to me (well, relatively - it has been observed I have been playing with the idea in and out for some more time) is entering this maze of twisty little passages, all alike called Drupal. What can I say? I'm quite surprised by it. It is such a reach CMS, and so twistable for almost-anything, that it still defeats me.

I've been asked (ordered? pushed?) to propose a complete plan to replace my Institute's current static-and-butt-ugly-HTML site with something dynamic and manageable, so, of course, this last week has been an intensive Drupal crash-course for me.

Drupal itself is quite complex, yes, and I thought I had it mostly mastered for the trivial tasks. But then, I started looking for some I-thought-quite-simple extra thingies - And I started discovering its user-contributed modules. I've been having quite a bit of fun with them, and as I hate messing up my clean and nice installation, I've even set up a Drupal5 modules APT repository for Etch, where I'm putting the modules as I process them.

As I'm really not into PHP, and I still lack enough of the Drupal framework understanding to really step forward and become responsable for them, I'm not yet even suggesting packaging them for Debian - but a time might come where I upload them as well ;-)

BTW, in case somebody is wandering about this Jaws-to-Drupal scripty I mentioned that other time: It basically works for blog entries (as you can see in my test site - barring some trivial latin1-UTF discrepancies). I have not yet migrated because I'm also trying to migrate my Phoo photo galleries to Acidfree albums... And it's quite a more challenging task than just migrating blog+comments. But soon, I hope - I have a bit more time than in the last weeks to be able to play with it. Anyway, here it is as it is.