the life of a sysadmin.

Tue Dec 30 11:47:56 PST 2008

A few quick notes about building Fedora Directory Server RPMs for CentOS:

You need to download and install fedora-ds-dsgw, as it's required by fedora-ds; this has been omitted from the list o' SRPMs to install.
You need to patch the spec file for fedora-ds itself. It requires fedora-ds-admin-console. That package was renamed in 1.1.2 from fedora-admin-console, but updated SRPMs for the renamed package were not provided (at least, they're not there as of today) for Fedora 6 (which is what's used to build for CentOS 5.2). Given the age of F6, it seems unlikely that updated SRPMs will be provided, so the simplest thing is to edit the spec file.
Also, the mmr.pl script mentioned here needs to be modified so that $instance_dir points to /etc/dirsrv, not /etc/fedora-ds.

(Partly a memo to myself, and partly to help anyone in the same boat; edits have been disabled in the FDS wiki, so I can't add this right now.)

(permalink) (comments)

Fri Dec 12 11:41:46 PST 2008

Tuesday: youngest son (8 months old) up at 5:30am teething.

Wednesday: youngest son up at 5:15am teething.

Thursday: youngest son up at 5:30am teething. I'm so tired I go to bed at 8:30pm and fall asleep immediately.

Friday: youngest son up at 4:45am teething. At 5:45am he goes back to sleep. At 6am my phone tells me the DNS server at work is down; I can't raise it. I restore backed up zone files to a spare Xen instance (hurrah!), give it the DNS server's IP address and head into work. I restart the machine and shut down the Xen instance; can't figure out why the machine shut down in the first place. Then I discover a replication problem between two of our LDAP servers which is resulting in random bounced email for a newly created account.

I want to go home now. But there's a Very Important Meeting(tm) at 1pm, and I can't leave before then.

(permalink) (comments)

Thu Dec 4 11:34:28 PST 2008

The Internet Storm Center writes about a new variant on malware that messes with your DNS: it installs a rogue DHCP server.

While not too sophisticated, the whole attack is very
interesting. First, it's about a race between the rogue DHCP server
and the legitimate one. Second, once a machine has been poisoned it is
impossible to detect how it actually got poisoned in the first place -
you will have to analyze network traffic to see the MAC address of
thoese DHCP Offer packets to find out where the infected machine
actually is.

In other news…all $job_2's new machines are set up and running. Kickstart is very nice…I really wish Debian had something similar; FAI is lovely, but Kickstart has the lovely feature of taking a hand-done installation you've just finished and turning that into a config file for a hands-off version. That saves a huge amount of time.

Next up: turn nscd back on (forgot I'd left it off for debugging LDAP 'til a simple find -exec chown was taking 10 minutes to finish); relabel the machines with their new names; commit the documentation I've been piecing together on my laptop; open up to others in the group; look at either moving the LDAP server over to the server room, or setting up a slave over there.

(permalink) (comments)

Thu Nov 27 10:59:10 PST 2008

From a list of known issues with installation of Office 2008 for Mac. Number one:

Office 2008 updates cannot be installed if the Microsoft Office 2008
folder was moved, renamed, or modified

Office Installer installs Microsoft Office 2008 for Mac in the
Applications folder. If you move the Microsoft Office 2008 folder to
another location on your computer, or if you rename or modify any of
the files in the Microsoft Office 2008 folder, you cannot install
product updates. To correct this issue so that you can install product
updates, drag the Microsoft Office 2008 folder to the Trash, and then
reinstall Office 2008 from your original installation disk.

Ah, hard-coded paths.

Number two:

I can't download the volume license version of Office 2008 for Mac by
using Safari

Cause: Downloading the volume license version of Microsoft Office 2008
for Mac is unsuccessful when you use the Safari browser.
Solution: We recommend that you use the latest version of Mozilla
Firefox Web browser ( MozillaClick this link to open a browser
window.http://www.mozilla.com) to download the volume license versions
of the Microsoft Office 2008 for Mac suite or stand-alone
applications.

That said, it turns out you don't need a license key for a volume copy of Mac Office 2008.

And now you know the rest of the story.

(permalink) (comments)

Sun Nov 23 21:06:12 PST 2008

I just spent the weekend (well, like an hour a day…kids, life, you know how it is) trying to track down why a bunch of new CentOS 5.2 installs at $job_2 couldn't pipe:

$ ls foo
foo
$ ls | grep foo
$ echo $?
141

(Actually, I didn't think to look at the error code 'til someone else pointed it out…141 turns out to be SIGIPE) In the end, it would have been quicker if I'd simply searched for the first thing I saw when logging in:

-bash: [: =: unary operator expected
-bash: [: -le: unary operator expected

This was particularly aggravating to track down because not every machine was doing this, and no matter what I thought to look at (/etc contents, /tmp permissions (those have a habit of going wonky on me for some reason), SELinux) I couldn't figure out what was different.

Turned out to be an upstream bug in nss_ldap. (The Bugzilla entry makes for some interesting reading, to be sure…) And I didn't see it on each machine because I hadn't upgraded after installation on all machines. (They're not yet in production, and I'm working on getting my kickstart straight.)

Man, it was gratifying to upgrade nss_ldap and see the problem go away…

(permalink) (comments)

Thu Nov 20 11:47:14 PST 2008

Old news by now, but I just got pointed to Dave considered harmful by a posting to the SAGE mailing list. Kudos to Sun for the full and thoughtful explanation.

(permalink) (comments)

Wed Nov 19 13:03:27 PST 2008

I've since found a great deal more about multipath in Linux:

The trick was to search for "multipath" and "fstab".

Also, I contacted the installer from Sun who worked on our new machines, and he told me that the multipath driver download was lost during an upgrade of the download page; they're working on it, but in the meantime he's sent me a copy of the driver. Sweet!

(permalink) (comments)

Wed Nov 19 09:01:16 PST 2008

Just now from the window, over the sound of a stupid high-pressure washer, I heard a Canada goose fly by, honking its head off.

(permalink) (comments)

Tue Nov 18 20:21:26 PST 2008

I'm in the process of setting up a bunch of new servers for $job_2. All but one are CentOS 5.2, kickstart installed and managed with cfengine. This is the third time I've goen thorugh a cfengine setup, and it always feels like starting from scratch each time. It seems — and I'm not at all sure this is fair or accurate — that each time I set up one of these systems, there's a lot that I've lost from the last time and have to relearn. I'm fortunate this time that I can refer to $job_1's setup to see how I did things last time, but if I didn't have that I'd be significantly further behind than I am.

I'm not sure what the solution is. Part of me thinks I should just be more aggressive about taking notes, or committing stuff to a private repository, or writing it down here more; part of me thinks that this might be a clue that cfengine is too low-level for my head. It feels like when I was trying to learn C, and couldn't believe that I had to remember all this stuff just to print something, or read a file, or connect to another machine over the Internet. By contrast, Perl (or any other scripted language) was such a relief…just print, or open, or use the Net::Telnet module, or whatever. The details are there and they are important, sometimes very much so; that doesn't mean I want to learn more metallurgy every time I need a fork. (No, I don't think that metaphor's tortured; why do you ask?)

Another thing is that I'm trying to get multipath connections working for the first time. We've got two database servers, each of which is connected via dual SAS HBAs to outboard disk arrays. (I don't think anyone else calls them "outboard", but I like the sound of it. See this hard drive? It's outboard, baby!) The arrays are from Sun and come with drivers, but the documentation is confusing: it says it's available for RHEL 5 (aka CentOS 5), but the actual download says it's only for RHEL 4.

As a temporary respite, I'm trying to see if I can get these workign using Linux's own multipath daemon, and it's also confusing. The documentation for it is tough to track down, and I just don't understand the different device names: am I meant to put /dev/dm-2 in fstab, or /dev/mpath/mpath2p1? If the latter, why does the name sometimes change to the WWUID (/dev/mpath/$(cat /dev/random)) when I restart multipathd? (use_friendly_names is uncommented in the config file.) If the whole point of multipath is failover, why does this sequence:

touch /mnt/1
remove first cable
rm /mnt/1
replace first cable
touch /mnt/2
remove second cable
rm /mnt/2
replace second cable

(where /mnt is where I've got this array mounted, obvs) sometimes work, and sometimes end with "I/O error" being logged, and the filesystem being read-only? Is this the sort of thing that the Sun driver will fix? I can't find anything about this.

And I mentioned electrical problems. When we got our servers installed, the Sun guys told us they'd tripped breakers on the PDU and/or breakers in the room's electrical cabinet. Since it had a sign on it saying "100A", I figured we might be running up against power limtis — either in the room as a whole, if my figures were 'way out, or on individual PDUs. Turns out I was probably wrong: I missed the bit on the sign that said 3-phase, which means (deep breath) we probably have 3 x 100A power available (I think).

It's more complicated than that, because some of it is in 120V, some of it is in twist-lock 220V 30A circuits, and so on. But I should've checked before emailing the faculty member who, in a year or two, will be going into this room (we're there as guests of the department) and happens to sit on the facilities committee. He had asked how we were doing, so I sent him an email — nice, polite, and including a bit about how grateful we were for the room and the help of the local sysadmins (all of which is true).

I was under the impression that he was asking for info now, so that he could bring it up for action in a few months when we were out. Instead, two hours later when I'm swearing at multipath, in come the facilities manager and one of the sysadmins I was dealing with, looking to find out just how much power we were using anyhow. I apologized profusely, and they were very cool about it. But when the committee guy asks questions, people jump. I had not anticipated this. Welcome to University Politics 101. I emailed again and explained my mistake.

There are lots of remedial courses I could take. However, today I would most like to take "Electricity and wiring for sysadmins".

And on another note: Ack! My laptop's home partition is 93% full! How the hell did that happen?

And again: How did I not know about apt-file? This is perfect!

(Touch o' the hat to Tears For Fears and Steve Kemp; I'm moving closer every day to switching to Chronicle.)

Building Fedora Directory Server RPMS for CentOS

So tired

By which I mean scary

Because I am amused by childish things. Why do you ask?

Well, that took a long time to track down

Wow

So that's where they're keeping it

I love working at UBC

This is The Working Hour; we are paid by those who learn by our mistakes

How handy is that? I mean, are those?