So… yeah. About the service the last year. Let’s just say my professional life got really, really, REALLY busy which started to compete with my personal life. The professional life involved a substantial amount of travel which unfortunately never really overlapped with my (now) fiancée’s. Of my personal life I have the reduced time with my fiancée and my personal geeky projects. One of those projects is a shell server which I’ve been running since 1996. Thankfully it’s not running on 1996 hardware or OS, but in the grand scheme of things it may as well have been.
The server is some Compaq running FreeBSD 6.3. The kernel is old. Very old. So much that I can’t build many new and shiny things like a modern enough Perl to do O-Auth with a module among other things. Now the logical thing would be to upgrade, but the server is 2000 miles away and doing the upgrade to an 8 kernel meant going to 7 first. Not the most trivial of tasks nor timely. So I decided to call a mulligan and start from scratch.
The new server is a HP DL380 with a bunch of fast disk and an external JBOD array with some SATA disk. I got FreeBSD 9 going and went to town migrating services to new and shiny homes. Then I ran into a challenge with how to move mail. The issue is, in part, that shell users got local mail in a mbox format. Virtual users are in maildir format. In the new world mail won’t live where the shell users are. Anyhow, long story short the migration stalled. I got everything but the mail moved over which means this service became split. The web bits live in the happy new home while the actual functional bits that take in email and make the posts is on the old dilapidated server.
Then I lost a drive on the internal array (RAID5).
Then I lost a drive on the external array (zfs).
So here I was mid migration, up to my eyeballs with work, and the amount of time for my personal life shrinking. I had to make an unfortunate compromise of letting the service coast as-is and hope I’d be able to get my life back in order.
So here I am in a flat in Austin, TX after day 1 of the Circuit Of The Americas F1 US Grand Prix. My flight out here I had the foresight to dump the mail spools for the Caltrain account and then wrote up some perl to parse through the message headers and generate a CSV file to import into the DB. About an hour ago I completed that import, and about an hour before that I finally figured out what was wrong with the web site. Stupid AllowOverrides.
In case these details make you more frustrated because they’re admittedly not difficult sysadmin problems to fix consider this: I’m not a sysadmin. I’m a network engineer who plays a sysadmin on TV and I don’t get paid to do it. It’s all out of pocket expenses to maintain things.
Bottom line, however, is I failed the community, the service, and myself by letting things go as long as I did. While the system has been functional in this maintenance-mode period, many people have been (im)patiently waiting for their key to contribute to the system. All requests that I was able to identify have been generated and are in the DB. All that is left is to send the mass email with the keys. That is something I will do in the next day or so. My goal is for the 92 people to have their key for Monday’s commute.
Again, my sincerest apologies.