The specifics are this:

1. Lucid switched to upstart.  To put it succinctly: “upstart is a replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.”

2. Upstart does not like autofs.

Not to rant too much, but if you are going to replace init, please do it with something that does not require that every person installing various packages has to do the hacks I am about to point out.

Thankfully, there are a bunch of smart, technical people that run ubuntu systems.  On top of that, when things go wrong they complain and post bug reports.  After some quick searching I thought I had fixed the bug.  That was until upstart was updated a couple of weeks back and the boot problems started again with autofs.

The solution is similar to the one I originally implemented, per the suggestion of comment #15, but it works past the update that had broken autofs again.  In the /etc/init directory edit the autofs.conf file and add the following stanza directly after the pre-start script line.

statd_status=`status statd| cut -d, -f1`
while [ "$statd_status" != "statd start/running" ]; do
sleep 5
start statd
let i++; statd_status=`status statd| cut -d, -f1`
if [ $i -gt 10 ]; then
echo "statd startup failed"
fi
done

Once this is in you should be able to (re)start autofs.  Next time the machine is rebooted, autofs will have been started automatically.  Essentially this is the same hack as the one in the aforementioned comment, with the exception that the while loop causes the script to wait until it sees statd has successfully started.

Ok, back to finding more annoying things.

Enter WPMU.  We are not running just one blog, we are running hundreds and have thousands of users.  Listing out all the users via a SQL query would be a huge list, not to mention not at all representative of who is an author on this particular blog.  I did some looking around on the wordpress codex and found a couple of functions that I thought could be helpful.  The first one that came up was wp_list_authors.  This function just lists out all the authors for a particular blog – particularly helpful for WPMU sites.

Now the problem with wp_list_authors is that it just outputs the list of authors as a chunk of links, so you have to chop it up somehow since its not in the loop – yeah, we are doing all this outside of the loop.  The second issue is that this is all that it puts out – links to the author archive page.  No ids, no emails, nothing – its not like the_author_meta which gives you all kinds of nice stuff.

Ok, but at least we have something we can hack up, so I started in on it and this is what I came up with.

<?php
$allAuthorNames = explode(',',wp_list_authors('style=0&show_fullname=1&hide_empty=0&echo=0'));
foreach ( $allAuthorNames as $oneAuthorName ) { ?>
  <li>
  <?php
  $oneAuthorArray = explode(" ",$oneAuthorName);
  if (count($oneAuthorArray) > 1) {
    $oneAuthorLink = explode("/",$oneAuthorArray[2]);
    end($oneAuthorLink);
    $userData = get_userdatabylogin(prev($oneAuthorLink));
  } else {
    $userData = array("user_email" => "None");
  }
echo get_avatar($userData->user_email,$size='96',$default='');
?>
<?php echo ($oneAuthorName); ?>
</li>
<?php } ?>

Pardon my PHP, it sucks, but in any case it gets it done here at least.

Notice the nice function that gets it done? Oddly, there is not much documentation to the get_userdatabylogin function, but its a nice one.  Tie together wp_list_authors with get_userdatabylogin and you can get even more info than you can get from the_author_meta.

Now this code is by no means the finished product, but it does work and it is a nice way to get a full list of everything that the author has in their profile in the DB. At the moment I just used it to get the email address of the author I was iterating over, but the function dumps out the entire user DB row object. A bit dangerous I suspect, but useful.

Happy coding,  hope this saves you a bit of time.

Hmm.  No.  Wait, its for LDAP?  Damn.

With that in mind, following are the instructions on changing a password where your password probably has not been changed in a while.

Changing your ssh key password with ssh-keygen

The -p option requests changing the passphrase of a private key file instead of creating a new private key. The program will prompt for the file containing the private key, for the old passphrase, and twice for the new passphrase. Use -f {filename} option to specifies the filename of the key file. For example, change directory to .ssh:

$ cd .ssh

To change your ssh-key passphrase, enter:

$ ssh-keygen -f id_{rsa or dsa} -p

Turns out that the Linux PAM system administrators guide was not really that bad and had some very useful explanations of how things work.

One thing I was looking to find out what the new bracket “[]” syntax for PAM config files works. Most admins will probably have seen this example somewhere:

auth [success=1 default=ignore] pam_ldap.so
auth required pam_unix.so try_first_pass
auth required pam_permit.so

What is up with those brackets? Turns out its pretty simple. According to the docs, the value=action control values are just a way of telling PAM what to do with the result of the rule (In this case, the rule is go check pam_unix.so and tell me what it said).

The value for the control can be set to any of the return values of the rule – success being rather obvious, default meaning anything that I didnt explicitly write. The action for the control can be ignore, bad, die, ok, done, reset or a number – the number just means skip the next N rules (the PAM SAG explains the config syntax fully).

In the above case, PAM is being instructed, in the common-auth config file to check LDAP first, ignore its failure if it cant find a username:password pair in the directory, or skip the next rule if it does find the pair. If it fails, it falls back to local accounts, if it succeeds iit jumps to the pam_permit.so rule which just permits anything it gets (keep in mind that if any required rule fails, the whole block fails, so thats why pam_permit.so cant permit something that pam_unix.so fails).

Thats it – pretty thought out by those clever PAM people. I hope I’ve explained it well enough here to help out. For a more in depth read, check out the Linux PAM system administrators guide.

My – aww man, do I really have to read this manual – pain, your gain.

I’ve been installing a new mail server, the usual suspects involved, postfix + cyrus. In the past they have always played well together, and they continue to do so today. Both, I think, are excellent choices when going for mail processing and end user delivery.

However, in this installation instance, which, as we hit the brass tax, did not work quite right.

–

The installation was done on a normal intel server running ubuntu 6.06.1 LTS release, with nothing special installed – pretty much vanilla. One this that was being done out of the ordinary was using an LDAP backend for the whole thing to authenticate off of. However, that didnt, as things end up, causing any issues.

After everything was installed and tested by the truly faithful who volunteered to move their mail spools over to the new server; it did have new features, was a more powerful server, have updated code, so it was an upgrade from the aging existing server. The problems started showing up after the server was put under some load. Postfix started reporting odd bounced messages, with a most peculiar error.

Jul 18 14:52:48 HOST postfix/lmtp[6763]: 9E7451008B: to=<...>, orig_to=<...>,
relay=/var/run/cyrus/socket/lmtp[/var/run/cyrus/socket/lmtp], delay=1, status=bounced
(host /var/run/cyrus/socket/lmtp[/var/run/cyrus/socket/lmtp] said: 250 2.1.5 ok (in reply to DATA command))

Hey postfix, cyrus said OK and you bounced the email. Are you guys listening to each other?

So I went to google and searched the error and came up with two interesting theories, by other admins who were also having issues with the same thing. I must admit that I would be defensive too, but Wietse and Viktor got a little crabby. Anyway, one says this is the fault of postfix pipelining and the other says this is the fault of cyrus duplicate suppression. One of them was partially right in my opinion.

http://www.irbs.net/internet/info-cyrus/0609/0435.html

http://archives.neohapsis.com/archives/postfix/2007-02/0940.html

So rather than disable pipelining, which would affect performance for almost everything dealing with postfix (although I think I understand now that you can just limit it for one particular place withing postfix and let other clients use it) I went and messed with duplicate suppression.

In imapd.conf I set, per the docs, duplicatesuppression: 0. Restarted cyrus, postfix for good measure and yay it seemed to be fixed. However while under load, the problem came back. Boo. I went ahead and reenabled duplicate suppression in cyrus, it was something that I wanted to keep anyway, however I knew that it had something to do with it so I kept digging in that general vicinity.

Turns out that Jon was right (in the above neohapsis post) and was duplicate suppression causing the problem, but not as he thought. The error that cyrus reported back in debug was the key – cyrus kept reporting how many lockers there were on the duplicate db, and that was causing postfix to abort. Lockers it turns out, related more to the db than to cyrus duplicate delivery.

In this case, the default setting, cyrus was using berkeley db as the backend for the duplicate database. I switched this setting (duplicate_db: skiplist) over to skiplist in imapd.conf and putting the server under load while watching the logs: the errors did not reappear.

Doing some more research on this, I found out that BDB 4.2 has a bug in it (which dapper is running and which cyrus is compiled with). Recompiling cyrus with BDB 4.3 or switching the duplicate suppression to use something that does not involve BDB 4.2 fixed the issue, even under load. Yay!

My pain, your gain.