Al Hoang

Awhile back I was noticing I was definitely losing emails. As one can might imagine, this is a scary experience since this brings into doubt if the mail system under use is doing something funny to the mail.

My first place to look was in the mail logs for the SMTP server and other associated daemons. However, I saw nothing in the maillogs which was not a very comforting thought.

After more investigation I would notice empty files like this every once in awhile…

~/Maildir)  ls -la new/
total 4
drwx------   2 al  al   512 Jun  8 00:25 .
drwx------  69 al  al  2048 Jun  8 00:25 ..
-rw-------   1 al  al     0 Jun  8 00:22 1244388142.30600_.myserver.net

This gave me more clues on where to look next. So next I looked in my Procmail logs for this particular mail id and noticed the process handling this message was killed by SIGPIPE

procmail: Executing "/usr/local/bin/spamassassin"
[84028] warn: spamassassin: killed by SIGPIPE
procmail: [84026] Tue Apr 14 21:45:26 2009

Googling dug up the following links that explain it all:

 http://www.nabble.com/Zero-exit-code-aft…
 https://issues.apache.org/SpamAssassin/s…

Verdict:
Upgrade Spamassassin

Since I have upgraded Spamassassin, the zero byte email mystery has resolved itself.

I have a love-hate affair with Capistrano. It is a great tool if you are a Ruby person and need to do something NOW on a bunch of machines. But the docs are in a constant state of suck from my point of view.

The Capify.org website helps for remembering the ’simple’ details on what Capistrano can do. But where I waste a lot of my time is asking questions like, “How do I set the Capistano SSH gateway to a non-standard port?”. Luckily, Capistrano is written in Ruby so it is easy enough to glance through the code and finally find out where it is but this is why good tech docs exist. To give enough context to answer those questions.

To answer my own question, below is a snippet you can add to your capfile to use a non-standard port if you need to deploy through a SSH gateway that lives on a non-standard port

# Add this to your Capfile
# This sets the SSH gateway to a machine called mysshgateway.com on port 22222
set :gateway, ‘mysshgateway.com:22222′

With the announcement of OpenSolaris 2009.06 I thought it would be appropriate to blog a little about a tool I had been writing to help myself play with Zones a bit easier.

My overall goals were the following:

• Have each zone configured with its own virtual NIC (Crossbow)
• Allow easy creation of zones without having to type zonecfg crap over and over again
• Make it a stepping stone to automatically creating zones
• See how well ipkg branded Zones work
• Allow a Zone to get its IP and DNS configuration from DHCP

I had tried going through tutorials that I found on the web (See references below) for setting up Zones but sadly none of them worked to my frustration. After a lot of experimentation I finally pieced together a way to create zones quickly and (almost) automatically for simple configurations.

Howto

1. Create a template zone that will be used as the main clone Zone
2. Download setup-zone-exclusive.sh and modify lines 34-35 to match the name of your template zone and the real interface you want the zones to bind to
3. Download the DHCP event hook script from here and name it dhcp-client-event.sh if you want DHCP configuration
4. Run setup-zone-exclusive with the zonename and the virtual nic interface that you want

In more detail here are the steps below

First create a template zone (I call it barebones here)

# Create /zones as its own ZFS filesystem
$ pfexec zfs create rpool/zones
$ pfexec zfs set mountpoint=/zones rpool/zones
$ pfexec zfs create rpool/zones/barebones
$ pfexec chmod 0700 /zones/barebones
$ pfexec dladm create-vnic -l $REAL_IF vnic0
$ pfexec zonecfg -z barebones
barebones: No such zone configured
Use ‘create’ to begin configuring a new zone.
zonecfg:barebones> create
zonecfg:barebones> set zonepath=/zones/barebones
zonecfg:barebones> set ip-type=exclusive
zonecfg:barebones> add net
zonecfg:barebones:net> set physical=vnic0
zonecfg:barebones:net> end
zonecfg:barebones> exit

$ pfexec zoneadm -z barebones install

Get the script

I would suggest you create a project directory to hold things such as zonecreations.

Download from Github gists here. Name it setup-zone-exclusive.sh. Don’t forget to chmod +x the file so you can execute it

Download the DHCP event hook script

You can get that here. Make sure this script is in the same directory as wherever you saved setup-zone-exclusive.sh

Create a zone

You can now create zones like this:

cd zonecreations
pfexec ./setup-zone-exclusive.sh mycoolnewzone virtualnic1

Have fun!

Update: Fixed an error in the example for using dladm. It should be correct now. Thanks!

References

Downloads

http://gist.github.com/122220 (setup-zone-exclusive.sh)
A DHCP event script to make sure DNS is configured when DHCP acquires an IP

Older docs on setting up Zones on Solaris

How to use sysidcfg file in OpenSolaris 2008.11
Internal Zone Configuration docs
Performing the Initial Zone configuration
Preconfiguring with sysidcfg file
OpenSolaris FAQ on sysidcfg
Ben Rockwood’s blogpost on Zone creation
About /etc/.UNCONFIGURED

Helpful for understanding Zones and Crossbow

Crossbow on vnics

Finding out that there is a change in policy for setting root_password in sysidcfg files

PASSREQ is enforced
zlogin failure after zone setup

The following helped in understanding the role of IPS and ipkg inside a non-global Zone

Updating Zones in OpenSolaris 2008.x
A field guide to Zones in OpenSolaris 2008.05
OpenSolaris forum on sysidcfg and Zones

Tom has a great summary on why.

One challenge for the cowboy sys admin is on how to keep oneself engaged while making their job basically… a walk in the park.

One thing I have found helpful in creating lists is to be dogmatic about writing docs as you are doing something somewhere, anywhere and collect all of this later. (You are writing documentation as you do your job, aren’t you?)

Read More

Seems someone else ran into issues while trying to deploy god.

While, I don’t think god sucks I definitely don’t endorse it. At this point I would only use it under the following conditions:

• Need for a process monitor tool with more dynamic configuration setups. This is where god really shines against monit’s simpler understanding of what process management is about.
• The host that needs monitoring can easily spare at least 16MB for a monitoring process. See below on why.
• I really want an all Ruby solution for all the tools in a system

In general, I am into the whole ‘It is Open Source. If you’re having issues, fix it’ deal so I am not nearly as angry sounding as Brad is about god. However, after having issues with god, I switched to monit for simple process monitoring and restarting. I had far less troubles and got on with other tasks that I considered more important than perfection in a process monitoring system.

For those that are curious here are the issues that I ran into with god:

• Daemonized Ruby took at least 8MB of RAM for the monitoring process. With RAM the way it is, this is not as big a deal. However, if you are trying to get by on a 128MB VPS host every kilobyte counts.
• God itself had issues just randomly dying after some time. Tom promptly fixed it after it was reported and that was great. However, it was a little disappointing that a monitoring process just died.
• Sparse documentation compared to monit’s. Then again this is typical from many Ruby projects and luckily Ruby code is readable enough
• Digging up known issues for god required noodling through groups, forums, and blog posts. Would have been nice to just have a friggin’ FAQ like other sys admin-targeted software I have seen.

I also DO agree as has been said in the comments on Brad’s post that it is the responsibility of the deployer of software to handle the issues with whatever they deploy and just deal with it. The reason I say this is because I fell for the hyped up description of god in the beginning and ultimately paid the price when it sucked up my time. I dealt with it but definitely am less impressed with overhyped marketing descriptions of software these days. Personally, I am not a fan of that type of marketing for software since it seems a little disingenuous to me. But that is just me.