Gunnar Wolf

The release of OpenSSH 5.4 was announced today. Its announced features include many small improvements, in usability and in crypto strength.

One of my favorite tricks using ssh is what Ganneff named ssh jumphosts – Many (most?) of my machines are not directly accessible from across the firewall, so the ability to specify in the configuration files where to jump through is most welcome. Well, with this "netcat mode" it will be much clearer to read and less of a hack… Of course, it loses a bit of the hackish æsthetic value, but becomes easier!

(yes, this post is basically a marker so I remember about it — But others might find it interesting)

Nobody cares about me, I thought. Whatever I say is just like throwing a bottle to the infinite ocean.

No comments, no hopes of getting any, for several days. Weeks maybe? Not even the spammers cared about me.

Until I read this mail, by Thijs Kinkhorst commenting to my yesterday post:

(…)
(BTW, I was unable to comment on your blog - couldn't even read one letter of the CAPTCHA...)

And, yes, Drupal module «captcha» introduced in its 2.1 release (January 2) feature #571344: Mix multiple fonts.

Only... no fonts were selected. Grah.

New guidelines for periodic publications' websites at my University favor the different journals we have to use a standardized system — And it makes quite a bit of sense. It is quite hard to explain to the people I work with that the content is not only meant to be consumed by humans, but also by other systems; the reasons behind rich content tagging and deep hierarchies for what they would just see as a list of words (think list of authors for an article, list of keywords, and so on). After all, aggregator databases such as Latindex and SciELO have achieved getting this understanding through.

And I must be quite grateful, as the University's guidelines point to what appears to be a very well-thought and thorough system, the Open Journal Systems by the Public Knowledge Project, co-funded by several well-regarded universities. OJS is a GPL-2-covered PHP bundle.

Anyway… I am very glad at least one of my Institute's journal accepted the challenge and decided to go OJS. I know I will quite probably be administering this system long-term. And, being as snobbish as I am, I know I loathe anything installed in my machines that is not either developed by myself or comes in a Debian package. So, as it was not packaged, I made the package ☺

Note that I am still not filing an ITP (which means, I have not yet decided whether I will upload this to Debian) because I want first to make sure I do have the needed long-term commitment — Besides, I am by far not a PHP person, and being responsible for a package… Carries a nontrivial weight. Still, you might be interested in getting it. If you are interested, you can either download the .deb package or add it to your apt repositories (and stay updated with any new releases), by adding this to your /etc/apt/sources.list:

Note: My packaging has still a small bug: The installer fails to create the PostgreSQL database. The MySQL database works fine. I will look into it soon

So far, I am quite impressed with this program's functionality and the depth/quality of its (online) documentation. Besides, its usage statistics speak for themselves:

So, it is quite possible I will be uploading this into Debian in a couple of weeks (hopefully in time to be considered for Squeeze). The reasons I am making it available in my personal repository now is:

• I want to make it available for other Debian- and Ubuntu- users in my University, as I am sure several people will be installing it soon. And after apt-getting it, it is just ready to be used right away.
• As I said, I am no PHP guy. So if you want to criticize my packaging (and even my minor patch, fixing a silly detail that comes from upstream's bundling of several PHP and Javascript libraries, and those libraries' authors not sticking to a published API in a well-distributed version), please go ahead!

Every now and then, people ask me why Debian? Why, among so many projects to choose from, I first liked, then got into, and finally I got committed into Debian, and not anything else?

Of course, one of the main points —back in 2000-2001 when I started using it, and still to this very day— is a strong identification with the ideological side. Yes, I am a strong Free Software believer, and Debian is what best suites my ideology.

Still, I did not only get into Debian because of this — And I was reminded about this by an article in this month's Usenix ;login: magazine: An anecdotal piece by Thomas A. Limoncelli titled Hey! I have to install and maintain this crap too, ya know! (article requires ;login: subscription, but I'll be glad to share it with whoever requests it to me — I have of course no permission to openly put it here in whole online. Yes, I am expressly sending a copy of this text to the author, I will update this if/when I hear from him) [update] The author has kindly allowed me to redistribute his article's PDF — Download it here.

Before anything else… I'll go on a short digression: I am writing a bit regarding the Free Software participants' culture, and this is a trait I love about it: The lack of formality. Even though ;login: (and Usenix as a whole) is not exactly Free Software, it runs quite close to it), it is a well regarded magazine (and association) with an academic format and good (not deep or highly theoretical, but good) contents. Still, it is quite usual to see titles as informal and inviting as this one. And it happens not only here — I have been fearing having to explain at work, over and over, why I have requesting permissions to go to Yet Another Perl Conference, Festival de Software Libre or DebCamp, tagging them as academic settings. Or why I am wasting our library's resources on buying cookbooks, recipes and similar material on the most strange-sounding subjects.

Anyway, back on track… This article I found refers to the lack of value given to the system administrator's time when selling or purchasing (or more in general, as it happens also in Free Software, when offering or adopting) a product. Quoting Thomas:

A person purchasing a product is focused on the features and benefits and the salesperson is focused on closing the deal. If the topic of installation does come up, a user thinks, “Who cares! My sysadmin will install it for me!” as if such services are free. Ironically, it is the same non-technical executive who dismisses installation and upkeep as if they are “free” who might complain that IT costs are too high and go on a quest to kill IT spending. But I digress.

I can understand why a product might be difficult to install. It is hard enough to write software, and with the shortage of software developers it seems perfectly reasonable that the installation script becomes an afterthought, possibly given to a low-ranking developer. The person purchasing the product usually requires certain features, and ease of installation is not a consideration during the procurement process. However, my ability to install a product affects my willingness to purchase more of the product.

Thomas goes on to explain his experience with Silicon Graphics, how Irix was so great regarding install automation and how they blew it when switching to Windows NT; talks very briefly about IBM AIX's smit, a very nifty sysadmin aid which is basically a point-and-click interface to system administration with the very nice extra that allows you to view the commands smit executes to perform a given action (and then you can copy into a script and send over to your hundreds of AIX machines)… Incidentally, by the time I started digging out of what became the RedHat mess of the late 1990s and passed briefly through OpenBSD on my way to Debian enlightenment, I was temporarily the sysadmin for an AIX machine — And I too loved this Smit approach, having it as the ultimate pedagogical tool you could ever find.

Anyway, I won't comment and paraphrase the full article. I'll just point out to the fact that… this was what ultimately sold me into Debian. The fact that I could just install anything and (by far) most of the times it will be configured and ready to use. Debian made my life so much easier! As a sysadmin, I didn't have to download, browse documentation, scratch head, redo from start until I got a package working — Just apt-get into it, and I'd be set. Of course, one of the bits I learnt back then was that Debian was for lazy people — Everything works in a certain way. Policy is enforced throughout.

So as a sysadmin, I should better get well acquinted with the Debian policy and know it by heart. In order to be able to enjoy my laziness, I should read it and study it. And so I did, and fell in love. And that is where my journey into becoming a Debian Developer started.

Why am I talking so nostalgic here? Because I got this magazine on the mail just last weekend… And coincidentally, I also got bug report #551258 — I packaged and uploaded the Haml Ruby library (Gem, as the Rubyists would call it). Haml is a great, succint markup language which makes HTML generation less of a mess. It is even fun and amazing to write Haml, and the result is always nicely formatted, valid HTML! And well, one of Haml's components is haml-elisp, the Emacs Lisp major mode to do proper syntax highlighting in Haml files.

Of course, I am an Emacs guy (and have been for over 25 years), so I had to package it. But I don't do Emacs Lisp! So I just stuffed the file in its (supposed) place, copying some stuff over from other Emacs packages. During DebConf, I got the very valuable help of Axel Beckert to fix a simple bug which prevented my package from properly being installed, and thought I was basically done with it. I was happy just to add this to my ~/.emacs and get over with it:

(require 'haml-mode)
(add-to-list 'auto-mode-alist '("\\.haml$" . haml-mode))
(require 'sass-mode)
(add-to-list 'auto-mode-alist '("\\.sass$" . sass-mode))

However… As Mike Castleman points out: This requires manual intervention. So it is not the Debian Way!

Reading Mike's bug report, and reading Thomas' article, made me realize I was dilluting something I held so dearly as to commit myself to the best Free Software-based distribution out there. And the solution, of course, was very simple: Debian allows us to be very lazy, not only as sysadmins, but as Debian packagers. Just drop this (simplified version) as $pkgroot/debian/haml-elisp.emacsen.startup and you are set!

(let ((package-dir (concat "/usr/share/"
                           (symbol-name flavor)
                           "/site-lisp/haml-elisp")))
;; If package-dir does not exist, the haml-mode package must have
;; removed but not purged, and we should skip the setup.
  (when (file-directory-p package-dir)
    ;; Use debian-pkg-add-load-path-item per §9 of debian emacs subpolicy
    (debian-pkg-add-load-path-item package-dir )
    (autoload 'haml-mode "haml-mode"
      "Major mode for editing haml-mode files." t)
    (add-to-list 'auto-mode-alist '("\\.haml\\'" . haml-mode))
    ;; The same package provides HAML and SASS modes in the same
    ;; directory - So repeat only the last two instructions for sass
    (autoload 'sass-mode "sass-mode"
      "Major mode for editing sass-mode files." t)
    (add-to-list 'auto-mode-alist '("\\.sass\\'" . sass-mode))
    ))

This will make the package just work as soon as it is installed, with no manual intervention required from the user. And it does not, contrary to what I feared, bloat up Emacs — Adding it to the auto-mode-alist leaves it as known to Emacs, but is not loaded or compiled unless it is required.

Deepest thanks to both of you! (and of course, thanks also to Manoj, for pointing out at the right spells in emacs-land)

Humm... Has anybody else seen a pattern like this?

I am getting a flurry of root login attempts at my main server at the University since yesterday 7:30AM (GMT-5). Now, from the machines I run in the 132.248.0.0/16 network (UNAM), only two listen to the world with ssh at port 22 — And yes, it is a very large network, but I am only getting this pattern on one of them (they are on different subnets, quite far apart). They are all attempting to log in as root, with a frequency that varies wildly, but is consistently over three times a minute right now. This is a sample of what I get in my logs:

[update] Logs omitted from blog post, as it is too wide and breaks displays for most users. You can download the log file instead.

Anyway… This comes from all over the world, and all the attempts are made as root (no attempts from unprivileged users). Of course, I have PermitRootLogin to no in /etc/ssh/sshd_config, but… I want to understand this as much as possible.

Initially it struck me that most of the attempts appeared to come from Europe (quite atypical for the usual botnet distribution), so I passed my logs through:

#!/usr/bin/perl
use Geo::IP;
use IO::File;
use strict;
my ($geoip, $fh, %by_ip, %by_ctry);
 
$fh = IO::File->new('/tmp/sshd_log');
$geoip=Geo::IP->new(GEOIP_STANDARD); 
while (my $lin = <$fh>) { next unless $lin =~ /rhost=(\S+)/; $by_ip{$1}++}; 
 
print "        Incidence by IP:\n", "Num  Ctry   IP\n", ('='x60),"\n";
 
for my $ip ( sort {$by_ip{$a} <=> $by_ip{$b}} keys %by_ip) {
    my $ctry = ($ip =~ /^[\d\.]+$/) ? 
	$geoip->country_code_by_addr($ip) : 
	$geoip->country_code_by_name($ip);
 
    $by_ctry{$ctry}++;
    printf "%3d  %3s  %s\n", $by_ip{$ip}, $ctry, $ip;
} 
 
print "        Incidence by country:\n", "Num  Country\n", "============\n";
map {printf "%3d  %s\n", $by_ctry{$_}, $_}
    sort {$by_ctry{$b} <=> $by_ctry{$a}} 
    keys(%by_ctry);

The top countries (where the number of attempts ≥ 5) are:

104  CN
 78  US
 58  BR
 49  DE
 43  PL
 20  ES
 20  IN
 19  RU
 17  CO
 17  UA
 16  IT
 13  AR
 12  ZA
 10  CA
 10  CH
  8  GB
  8  AT
  8  JP
  8  FR
  7  KR
  7  HK
  7  PE
  7  ID
  6  PT
  5  CZ
  5  AU
  5  BE
  5  SE
  5  RO
  5  MX

I am attaching to this post the relevant log (filtering out all the information I could regarding legitimate users) as well as the full output. In case somebody has seen this kind of wormish botnetish behaviour lately… please comment.

[Update] I have tried getting some data regarding the attacking machines, running a simple nmap -O -vv against a random sample (five machines, I hope I am not being too agressive in anybody's eyes). They all seem to be running some flavor of Linux (according to the OS fingerprinting), but the list of open ports varies wildly — I have seen the following:

Not shown: 979 closed ports
PORT      STATE    SERVICE
21/tcp    open     ftp
22/tcp    open     ssh
23/tcp    open     telnet
111/tcp   open     rpcbind
135/tcp   filtered msrpc
139/tcp   filtered netbios-ssn
445/tcp   filtered microsoft-ds
593/tcp   filtered http-rpc-epmap
992/tcp   open     telnets
1025/tcp  filtered NFS-or-IIS
1080/tcp  filtered socks
1433/tcp  filtered ms-sql-s
1434/tcp  filtered ms-sql-m
2049/tcp  open     nfs
4242/tcp  filtered unknown
4444/tcp  filtered krb524
6346/tcp  filtered gnutella
6881/tcp  filtered bittorrent-tracker
8888/tcp  filtered sun-answerbook
10000/tcp open     snet-sensor-mgmt
45100/tcp filtered unknown
Device type: general purpose|WAP|PBX
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (96%), (…)
 
 
Not shown: 993 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
25/tcp   open   smtp
80/tcp   open   http
443/tcp  open   https
444/tcp  open   snpp
3389/tcp open   ms-term-serv
4125/tcp closed rww
Device type: general purpose|phone|WAP|router
Running (JUST GUESSING) : Linux 2.6.X (91%), (…)
 
Not shown: 994 filtered ports
PORT    STATE  SERVICE
22/tcp  open   ssh
25/tcp  closed smtp
53/tcp  closed domain
80/tcp  open   http
113/tcp closed auth
443/tcp closed https
Device type: general purpose
Running (JUST GUESSING) : Linux 2.6.X (90%)
OS fingerprint not ideal because: Didn't receive UDP response. Please try again with -sSU
Aggressive OS guesses: Linux 2.6.15 - 2.6.26 (90%), Linux 2.6.23 (89%), (…)
 
Not shown: 982 closed ports
PORT      STATE    SERVICE
21/tcp    open     ftp
22/tcp    open     ssh
37/tcp    open     time
80/tcp    open     http
113/tcp   open     auth
135/tcp   filtered msrpc
139/tcp   filtered netbios-ssn
445/tcp   filtered microsoft-ds
1025/tcp  filtered NFS-or-IIS
1080/tcp  filtered socks
1433/tcp  filtered ms-sql-s
1434/tcp  filtered ms-sql-m
4242/tcp  filtered unknown
4444/tcp  filtered krb524
6346/tcp  filtered gnutella
6881/tcp  filtered bittorrent-tracker
8888/tcp  filtered sun-answerbook
45100/tcp filtered unknown
Device type: general purpose|WAP|broadband router
Running (JUST GUESSING) : Linux 2.6.X|2.4.X (95%), (…)
 
Not shown: 994 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
3389/tcp open  ms-term-serv
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: firewall|general purpose
Running: Linux 2.6.X
OS details: Smoothwall firewall (Linux 2.6.16.53), Linux 2.6.13 - 2.6.24, Linux 2.6.16

Of course, it strikes me that several among said machines seem to be Linuxes, but (appear to) run Microsoft services. Oh, and they also have P2P clients.

I met my friend Josef Daberning, who did his Austrian Social Service working with Drupal at the Casa de los Tres Mundos NGO, in Granada, Nicaragua, at the Central American Free Software Encounter, last May. He told me that, when going back to Austria, he would spend some days in Mexico, and wanted to give a workshop on Drupal.

The course has just started, and will take place today and tomorrow — You can follow the live stream at http://www.iiec.unam.mx:18000/drupal.ogg — The videos will be uploaded soon as well, I will post them on this same node.

This node will be used for whatever is needed to make public for people following the talk. As of right now, you can download his presentation — http://gwolf.org/files/gira-drupal.odp and http://gwolf.org/files/gira-drupal.pdf

[update](s):

• If you are following the stream and want to say something, connect via IRC to OFTC (irc.debian.org) and join the #edusol channel.
• The videos for the talks will be made available soon, licensed under CC-BY 3.0 (generic)
• If you want to come, you are more than welcome. There is still space! We are in Ciudad Universitaria, South of Mexico City. Instituto de Investigaciones Económicas - UNAM. We are starting the second session (16:00-10:00), tomorrow we have a third and final session (10:00-14:00).
• Of course,it is all over by now. I will be processing the videos and uploading them to Archive.org's Open Source Movies archive. The three videos (for the three sessions) are available!

Oops.

Yesterday (Sunday, 31/08/09) I far from any computer-like object for most of the day. When I got back home, of course, I promptly opened my laptop to check my mail — who knows what destiny might have for me in a 24 hour period? Maybe I won yet another fortune I have to cash in Nigeria? Maybe there is (GASP!) a new RC bug on one of my packages?

But no, my mail server didn't feel like answering to my ssh queries. The connection was established, but shut down before even sending the protocolary SSH-2.0-OpenSSH_5.1p1 string. Fearing an overload (after all, the little bugger is just a Mac Mini running in another room in my house), I tried to check (via Web) its Munin status — Apache didn't want to listen either. It answered, but got only access denied. Things started worrying me… But (silly me) not enough — The machine runs headless1, so I just danced the boring raising elephants song2.

Allowed for a couple of minutes for everything to settle, and tried to connect. Horror, now even pings didn't work!

So I ran to fetch my old, bulky and trusty monitor. Went back to the machine, plugged it in, switched it off and back on. Everything worked fine this time — At least appearingly. I opened up mutt and started happily reading mails, while trying to understand on another console what happened at 07:06 that didn't get logged anywhere and had the machine dead for basically all the day. And then, BRRRT-BRRRT-BRRRT, I started hearing the HDD seeking.

I was able to send a couple of mails, but decided to let the machine rest and... Will reduce its disk usage to an absolute minimum. Fortunately, I have already the machine meant to replace it — A much nicer, beefier iMac G5, waiting to be vacated from its data, task which has suddenly become prioritary.

So, in short: If you need to get in touch with me in the next day or two, don't count on my usual @gwolf.org mail, as it is down. I hope to be able to get the data out of the poor little bugger painlessly after it rests a bit. And I hope not to drown in a sea of mails after I get the replacement back online :-/

1. 1. for those not used to computerspeak: without a connected monitor
2. 2. For those following at home who don't understand how Raising Elephants Is So Utterly Boring, this is a (silly, stupid but useful) mnemotecnic for as-properly-as-possible restarting hung Linux systems: Hold AltGr (right-Alt) + SysRq (in reduced/notebook keyboards, Fn+SysRq), and type R E I S U B (leaving a couple of seconds between key, ideally waiting for all disk activity to settle between each). That means:

   unRaw

   Take control of keyboard back from X,

   tErminate

   Send SIGTERM to all processes, allowing them to terminate gracefully,

   kIll

   send SIGKILL to all processes, forcing them to terminate immediately,

   Sync

   Flush data to disk),

   Unmount

   Remount all filesystems read-only,

   reBoot.

   Well... Reboot the system

Ufff...

I have spent a couple of hours connected from Norman García's house, in Managua. Norman is most kindly hosting me at home for a couple of days before we leave (tomorrow) for Estelí, where the Central American Free Software Encounter will be held.

Now, the network feels really slow. However, it can sustain download rates of around 512Kbps, quite acceptable. Latency is what kills. But... I was stunned with mtr's results to my home server:

 Host                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. speedtouch.lan                                           0.0%    16  101.1  51.0   3.2 101.1  32.2
 2. 1-0-212-190.enitel.net.ni                               12.5%    16  210.6 156.3  21.4 376.4  91.7
 3. 17-83-62-200.enitel.net.ni                              86.7%    16   23.0  22.8  22.6  23.0   0.2
 4. 10.192.100.49                                           66.7%    16  219.7 166.1  30.9 283.0  94.4
 5. 10.192.4.89                                             86.7%    16  136.7 106.7  76.7 136.7  42.5
 6. 10.192.4.82                                             66.7%    16  125.2 130.2  42.2 233.5  72.0
 7. 10.192.4.46                                             66.7%    16  249.9 130.3  55.2 249.9  75.3
 8. 10.192.4.50                                             92.9%    15   41.3  41.3  41.3  41.3   0.0
 9. 10.192.4.54                                             66.7%    15   85.6 138.8  85.6 169.7  33.4
10. 10.192.4.86                                             64.3%    15  141.1 154.9  82.4 264.0  66.8
11. 10.192.5.17                                             78.6%    15   86.4 150.3  74.1 290.5 121.5
12. 10.192.5.21                                             86.7%    15  112.6  78.0  43.4 112.6  49.0
13. 10.192.3.17                                             57.1%    15  225.1 152.6  52.5 246.1  77.7
14. 10.192.3.9                                              53.8%    14   69.5 151.4  69.5 235.6  52.8
15. 10.192.3.5                                              84.6%    14  236.0 141.1  46.1 236.0 134.3
16. 10.192.3.30                                             71.4%    14  116.3 161.2  39.4 258.0 101.9
17. 10.192.3.34                                             84.6%    14   67.9  52.5  37.1  67.9  21.8
18. 10.192.2.129                                            76.9%    14  258.2 172.0 112.6 258.2  76.4
19. 10.192.2.241                                            61.5%    14  159.8 117.3  42.3 159.8  51.8
20. 10.192.6.13                                             78.6%    14  128.1 192.4 128.1 240.4  57.9
21. 10.192.2.141                                            92.3%    14   63.8  63.8  63.8  63.8   0.0
22. 10.192.2.45                                             58.3%    13  249.3 196.3 136.3 249.3  46.2
23. 10.192.2.41                                             91.7%    13  260.5 260.5 260.5 260.5   0.0
24. 10.192.2.34                                             69.2%    13  231.1 208.7  82.5 260.7  85.3
25. ???
26. inet-pue-fuertes-24-pos6-2.uninet.net.mx                63.6%    12  158.0 172.0 111.3 261.2  63.4
27. dsl-mex-quevedo-2-pos2-0.uninet.net.mx                  90.9%    12  161.0 161.0 161.0 161.0   0.0
28. ???

Please, somebody explain the basics of routing to Claro/Enitel. This just does not make any sense.

You might have noticed that during last week the Mexican Debian mirror, nisamox.fciencias.unam.mx (a.k.a. ftp.mx.debian.org, a.k.a. debian.unam.mx), went offline. The motherboard died on us, and Facultad de Ciencias was kind enough to give us a brand new one. So, excuse us for the blackout, but we are back – Meaner and badder than ever before!

Now, Sergio (nisamox's main admin) prefered to rebuild the whole mirror, as there was a shadow of doubt regarding the data integrity. So, rsync was pulling as fast as he could for the whole weekend (leading to some people scratching their heads regarding the 404 for the missing files; sorry, we should have left Apache shut down until the mirror was complete!). After three days of sustaining a 10-20Mbps download from the main mirrors, all 364GB of Debian are finally installed and –as you can clearly see– we are back to normality, with small, regular mirror pulses and a nice sustained 5-10Mbps (with some up to 40Mbps peaks — We have seen up to 100Mbps peaks in the past, and I doubt with the current network infrastructure we won't top that).

You can see we have currently plenty of disk space still to fill up. Among our plans is to host the most popular ISOs, which are a common request, and... What else? Well, ask us and we shall do so (quite probably).

So, if you switched away from ftp.mx.debian.org due to our downtime, readjust your mirror settings. Nisamox is back!

The phrase on this title is often attributed to Pancho Villa (1878-1923), Mexican Revolution leader. He had a fame of cruelty, killing suspects before even questioning them.
Today, it started as a very nice day. I had even time in the morning to find, fix, upload and send upstream a trivial bug in libgruff-ruby... At 11:00, I left the Institute as my father came to the city to do some paperwork... We sat having a cup of coffee in a restaurant near the office we had went to at around 12:00, and my phone rang.
And it was from work. That's never a good sign. My boss told me he was facing a massive virus infection, and decided to disconnect the firewall. I corrected him - that will do no good once the virus is in our system, if you want to disconnect anything, disconnect all of our switches.
Came back, and found him and my coworker stunned and not knowing what to do. He says, the antivirus alarm went off almost simultaneously on the two computers he had on his desk, and in few minutes over 15 computers all over the Institute were ill. The symptoms? Programs not showing up in the taskbar, copy/paste functionality b0rken, many programs misbehaved or just didn't open... They were grimly facing a complete recovery operation they have grown used to: The whole OS has become corrupted or destroyed, we will have to open the computer, extract the HD, install it elsewhere, back it up, reinstall OS and applications, restore the backup. Yes, I know too many extra steps are included here, but I have come to accept their ways of dealing with Windows. Nobody says dealing with Windows is fun. I like my work to be fun, so I stay clear of theirs.
I insisted on turning back one one of the switches, the one for the servers and my machine (and some more in the same physical area). OK'd. But they didn't want to switch on any other switch, so a traffic capture (tcpdump / wireshark) led nowhere - but at least it gave my my Google back.
They have configured the antivirus software we deploy to all of the Windows machines in such a way that it deletes upon sight any malware - And when they manually scan, they blindly hit Delete whenever anything is found as well. Of course, no infected binary was left alive for me to inspect, and the machines were dead. But I was able to glimpse at the name of the deleted file: rpcss.dll.
After googling a bit - Bliss! Joy! I found the answer. So here is the set of interactions, and how they led to this killing spree. Please remember I am a Windows newbie and speak just out of guesswork.

1. This is a fast-spreading virus. My friend Rubén at DGSCA suggest it might be related to this report submitted today; at Barrapunto there is a thread about another virus that appeared four days ago, infected 1.1 million Windows machines on its first day, and so far is around the ninth million. Update: Equivalent thread at Slashdot, for the Spanish-impaired.
2. The virus infects at least two copies of a system binary: %system32%\rpcss.dll and \Windows\ServicePackFiles\i386\rpcss.dll. Windows uses the second one to restore the first one in case it is damaged, if I understood correctly.
3. The antivirus does not detect the infection when the library files are written, but when they are linked, so it only spots it the next time %system32%\rpcss.dll is brought into memory.
4. This is a very common library - It takes care of, well, RPC. So, quite probably, this file will be linked again on the next program launch - or accessed when a running program requires anything not currently in RAM? Dunno. The thing is, the library gets linked.
5. The antivirus will happily tell you it has killed a threat! Your nice RPC library is now defunct. ¡Mátalo, luego virigüas!
6. So, of course, notifying the taskbar of a new window appearing, or clipboard actions, or whatnot will refuse to work.
7. Machine restart, full system scan requested. The antivirus finds de second copy of this library in the master directory (\Windows\ServicePackFiles\i386). The virus used this location so that Windows won't restore a clean version over it. But yes, it will fall again under the claws of the antivirus... I guess. Anyway, the antivirus offers to delete this file as well, and does so.
8. User is desperate. My coworkers are desperate. I am... mildly annoyed?

Once I found this line of thought... I went to a working machine, inserted my flash memory, and copied %system32%\rpcss.dll to it. Went back to a sick machine, and ran cmd. Then, it was just matter of copy f:\rpcss.dll c:\windows\system32, a simple reboot (it never hurts to reboot in Windows!), and problem solved!
Oh, as a side rant: I find it extremely annoying and sad that many people I know, sometimes with more experience as a computer operator/supporter than what I have of experience as a living human being, are so scared of using a command-line interface. They were dismayed at seeing no drag-and-drop and no copy/paste functionality were available! copy is not an option.
Anyway... Today was an experience on how a simple, mostly-harmless and quite-fertile virus is able to be terribly magnified by the presence of a trigger-happy antivirus.
Why won't they give themselves a chance to try something else? Say, GNU/Linux? :-/

Wouter insists that Ruby Gems are enough of an argument to keep Rails at a distance. Even though I agree with the basic claim and think that Gems are basically insane and sick, this should be taken a bit more under perspective.
We are blessed. We are blessed to have Debian, such a rich OS with such a great package management system, and with superb integration between so many packages. Blessed are also the users of most Free Software based distributions, as they share the advantages of systems growing with full consciousness of their interaction's benefits. However, integration with the rest of the world is not seamless.
Most scripting languages have their own infrastructure for managing the modules/libraries/pacakges/whatevers dependencies. Perl has CPAN, PHP has PEAR, Ruby has Gems... I do see Gems as the most obnoxious of them all, but the basics are the same.
The Rails crowd started being Unix-centric, but the Windows (and MacOS - they are no better, believe me, at least in this regard) world has exerted its pressure. Gems caters very well to their needs, but we do suffer the integration at the distro side.
The only sane way the propietary-minded people have managed to stay clear of the well known "DLL hell" is to ship everything a given program requires bundled together - that's the main reason for the bloat of lots of applications, and for the sloppiness of security support. Every application packager is responsible for shipping updated versions to any library it bundles in, except for the very basic core that the OS itself provides. That seems so annoyingly backwards to us that... it is unbelievable.
So, yes, Rails application trees often include Rails itself. For $DEITY sake, even I have grown used to working that way, as things tend to break under your nose otherwise. My proposal (which we talked over at DebConf, but have not pushed so far) is to support simultaneous versions of Rails installed in a Debian system (of course, via different packages), more or less in the way that simultaneous versions of Ruby, PHP or Python (and, in some limited fashion, Perl - Although Perl does not suffer from this incompatible bumps. Yay for Perl!) can be installed.
...And, yes, together with the pkg-ruby-extras team, I have been trying to -slowly, yes- package whatever modules we often use so they don't have to be included in Rails applications.
So far, the best way (although by far not optimal) I have found to limit this explosion of trees is to include most libraries in my Rails application trees as git submodule trees - If they are not explicitly downloaded, the systemwide libraries will be used.
Yes, the "ship the whole thing as a bundle" approach is quite annoying. However, at least I must acknowledge that it works better than the approach I took with previous (mod_perl-based) webapps I wrote... As relatively few people grok mod_perl, I ended up with quite some apps which were not installed by anybody but me. Rails is obnoxious... but seems to be parsable by the humanity at large.

Via Kees Cook (and sorry for the reiteration for people following along Planet Debian, thanks to Caspar Clemens: Recent (>= 5.1) versions of OpenSSH (found at least in Debian Lenny and Ubuntu Intrepid), have the VisualHostKey option. What does it do?

$ ssh -o VisualHostKey=yes 172.16.10.1
Host key fingerprint is db:7a:d8:a8:2e:41:a2:e5:51:e1:7f:d0:73:bd:85:bf
+--[ RSA 2048]----+
|    ..           |
|   ..  .   . .   |
|   .. . o . o .  |
|  + .. . o   +   |
| + +  . S   . .  |
|. . .  . o     . |
|     .  .+.   E  |
|    .   o.o      |
|     oo...       |
+-----------------+

Linux respaldos.local.iiec 2.6.26-1-vserver-amd64 #1 SMP Wed Oct 1 13:08:10 UTC 2008 x86_64

What does this mean? This ASCII-art graph represents your host's public key, which uniquely identifies (or at least, it better damn should uniquely identify!) it. This representation was added mainly because it is way easier to be able to visually record the shape of your most frequently used hosts' IDs than their fingerprint. If you connect from a foreign or untrusted machine (i.e. one that does not yet know your host's identity), make sure to run with this switch - it will protect you from somebody supplanting your server's identity.
Besides, it adds to the general kewlness factor, doesn't it? ;-)
To enable this behaviour by default, add the following to your /etc/ssh/ssh_config (or to your personal .ssh/config):

Host * 
  VisualHostKey yes

Now... What about publishing the list of the 32767 known-bad SSH keys? That'd make for a nice ASCII-art exhibit :-}

Several weeks ago, the people in charge of maintaining the Windows machines in my institute were desperate because of a series of virus outbreaks - Specially, as expected, in the public lab - but the whole network smell virulent. After seeing their desperation, I asked Rolman to help me come up with a solution. He suggested me to try replacing the Windows workstations by substituting local installations by a server having several virtual machines, all regenerated from a clean image every day, and exporting rdesktop sessions. He suggested using Xen for this, as it is the virtualization/paravirtualization solution until now best offered and supported by most Linux distributions (including, of course, RedHat, towards which he is biased, and Debian, towards I am... more than biased, even bent). So far, no hassle, right?
Of course, I could just stay clear of this mess, as everything related to Windows is off my hands... But in October, we will be renewing ~150 antivirus licences. I want to save that money by giving a better solution, even if part of that money gets translated to a big server.
Get the hardware
But problems soon arose. The first issue was hardware. Xen can act in its paravirtualization mode on basically any x86 machine - but it requires a patched guest kernel. That means, I can paravitualize many several different free OSs on just any computer I lay my hands on here, but Windows requires full- or hardware-assisted- virtualization. And, of course, only one of the over 300 computers we have (around 100 of which are recent enough for me to expect to be usable as a proof-of-concept for this) has a CPU with VT extensions - And I'm not going to de-comission my firewall to become a test server! ;-)
When software gets confused for hardware
So, I requested a Intel® Core™2 Quad Q9300 CPU, which I could just drop in any box with a fitting motherboard. But, of course, I'm not the only person requiring computer-related stuff. So, after pestering the people in charge for buying stuff on a daily basis for three weeks, the head of acquisitions came smiling to my office with a little box in his hands.
But no, it was not my Core 2 Quad CPU.
It was a box containing... Microsoft Visio. Yes, they spent their effort looking for the wrong computer-related thingy :-/ And meanwhile, Debconf 8 is getting nearer and nearer. Why does that matter? Because I have a deadline: By October, I want the institute to decide not to buy 150 antivirus licenses! Debconf will take some time off that target from me.
Anyway... The university vacations started on July 5. The first week of vacations I went to sweat my ass off at Monterrey, by Monday 14 I came back to my office, and that same day I finally got the box, together with two 2GB DIMMs.
Experiences with a nice looking potential disaster
Anyway, by Tuesday I got the CPU running, and a regular Debian install in place. A very nice workhorse: 5GB RAM, quad core CPU at 2.5GHz, 6MB cache (which seems to be split in two 3MB banks, each for two cores - but that's pure speculation from me). I installed Lenny (Debian testing), which is very soon going to freeze and by the time this becomes a production server will be very close to being a stable release, and I wanted to take advantage of the newest Xen administration tools. Of course, the installation was for AMD64 - Because 64 bitness is a terrible thing to waste.
But I started playing with Xen - And all kind of disasters stroke. First, although there is a Xen-enabled 2.6.25 Linux kernel, it is -686 only (i.e. no 64 bit support). Ok, install a second system on a second partition. Oh, but this kernel is only domU-able (this is, it will correctly run in a Xen paravirtualized host), but not dom0-able (it cannot act as a root domain). Grmbl.
So, get Etch's 2.6.18 AMD64 Xen-enabled kernel, and hope for the best. After all, up to this point, I was basically aware of many of the facts I mentioned (i.e. up to this point I did reinstall once, but not three times)... And I hoped the kernel team would have good news regarding a forward-port of the Xen dom0 patches to 2.6.25 - because losing dom0 support was IMO a big regression.
But quite on time, this revealing thread came up on the debian-devel mailing list. In short: Xen is a disaster. The Xen developers have done their work quite far away from the kernel developers, and the last decent synchronization that was made was in 2.6.18, over two years ago. Not surprisingly, enterprise-editions of other Linux distributions also ship that kernel version. There are some forward-patches, but current support in Xen is... Lacking, to say the least. From my POV, Xen's future in the Linux kernel looks bleakish.
Now, on the lightweight side...
Xen is also a bit too complicated - Of course, its role is also complicated as well, and it has a great deal of tunability. But I decided to keep a clean Lenny AMD64 install, and give KVM, the Kernel Virtual Machine a go. My first gripe? What a bad choice of name. Not only Google searches for KVM gives completely unrelated answers (to a name that's already well known, even in the same context, even in the same community).
KVM takes a much, much simpler approach to virtualization (both para- and full-): We don't need no stinkin' hypervisors. The kernel can just do that task. And then, kvm becomes just another almost-regular process. How nice!
In fact, KVM borrows so very much from qemu that it even refers to qemu's manpage for everything but two command-line switches.
Qemu is a completely different project, which gets to a very similar place but from the other extreme - Qemu started off as Bochs, a very slow but very useful multi-architecture emulator. Qemu started adding all kinds of optimizations, and it is nearly useful (i.e. I use it in my desktop whenever I need a W2K machine).
Instead of a heavyweight framework... KVM is just a modprobe away - Just ask Linux to modprobe kvm, and kvm -hda /path/to/your/hd/image gets you a working machine.
Anyway - I was immediatly happy with KVM. It took me a week to get a whole "lab" of 15 virtual computers (256MB RAM works surprisingly well for a regular XP install!) configured to start at boot time off a single master image over qcow images.
KVM's shortcomings
Xen has already been a long time in the enterprise, and has a nice suite of administrative tools. While Xen depends on having a configuration file for each host, KVM expects them to be passed at the command line. To get a bird-eye view of the system, xen has a load of utilities - KVM does not. And although RedHat's virt-manager is said to support KVM and qemu virtualization (besides its native Xen, of course), it falls short of what I need (i.e. it relies on a configuration file... which lacks expresivity to specify a snapshot-based HD image).
To my surprise, KVM has attained much of Xen's most amazing capabilities, such as the live migration. And although it's easier to just use fully virtualized devices (i.e. to use an emulation of the RTL8139 network card), as they require no drivers extraneous to the operating system, performance can be greatly enhanced by using the VirtIO devices. KVM is quickly evolving, and I predict it will largely overtake Xen's (and of course, vmware and others) places.
Where I am now
So... Well, those of us that adopt KVM and want to get it into production now will have some work of building the tools to gracefully manage and report it, it seems. I won't be touching much my setup until after Debconf, but so far I've done some work over Freddie Cash's kvmctl script. I'm submitting him some patches to make his script (IMHO) more reliable and automatizable (if you are interested, you can get my current version of the script as well). And... Starting September, I expect to start working on a control interface able to cover my other needs (such as distributing configuration to the terminals-to-be, or centrally managing the configurations).

Last week (July 7-13) was basically hell on Earth, for me and for the group that somehow got the name Cabras locas, of which I am part since I joined the National Pedagogical University, where I worked full-time 2003-2005.
It was, yes, the first of my officially three weeks of Summer holiday at IIEc-UNAM, so no problems here. So, why hell on Earth? Because we were in charge basically of anything related with information flow, retrieval and manipulation at the 11th International Congress on Mathematical Education, in Monterrey.
What we thought would basically be one or two days of hard work followed by six days of relaxed vacations (we had even planned to have an internal seminar, showing off the shiny stuff each of us is working on) became... A mind-boggling eight day experience where we worked over 12 hours a day on being human replacements for Google, SQL engines, full-text parsers, report generators, printer watchdogs, and in general lines, just a bunch of unhappy firemen, ready to be called off for whatever task was necessary.
We did have, of course, several calm periods every now and then. We even had to learn how to look busy while doing something compeltely unrelated (that would explain, for example, a couple of low-hanging bugs I fixed for Debian, or some dozens of lines of code I could get off my head).
But my advice for whoever reads this: Don't trust people with long database-handling experience. Specially when they insist that hand-capturing a thousand registers is preferrable (i.e. less error-prone) than parsing three separate databases and discarding duplicates. And, of course, specially when this person is your boss, which is enough of an argument to have it his way.

I think I should follow up on Victor's lament. Yes, we have a Rails application which works fine most of the time... But quite often, throws out a segmentation fault I just have been unable to pin-point. It might be related to rmagick, the only non-pure-Ruby component I am using (and I'm tempted to try minimagick instead, even if I prefer in-memory operations than on-disk, piping an image and slurping it again).
Victor came up with an easy script to check the server - but to reduce the impact it has (I was running a single Mongrel instance, which meant, whenever it dies the whole system becomes inaccessible for everybody; I replaced it with a mongrel_cluster of five processes, plus pound as a easy-to-use balancer which looks quite nice), the very simplistic and to-the-point script did no longer work.
Anyway... Ruby rocks ;-) I'm sharing this with you mostly because I am sure some readers will find more than one useful construct, not because it is precisely beautiful code. And besides, we should work on fixing the cause, not the consequence, of the bug! :)

#!/usr/bin/ruby
require 'yaml'
confdir = '/etc/mongrel-cluster/sites-enabled'
restart_cmd = '/etc/init.d/mongrel-cluster restart'
needs_restart = false
 
(Dir.open(confdir).entries - ['.', '..']).each do |site|
  conf = YAML.load_file "#{confdir}/#{site}"
  pid_location = [conf['cwd'], conf['pid_file']].join('/').gsub(/\.pid$/, '*.pid')
  pid_files = Dir.glob(pid_location)
 
  pid_files.each do |pidf|
    pid = File.read(pidf)
    begin
      Process.getpgid(pid.to_i)
    rescue Errno::ESRCH
      warn "Process #{pid} (cluster #{site}) is dead!"
      File.unlink pidf
      needs_restart = true
    end
  end
end
 
system(restart_cmd) if needs_restart