Al Hoang

Awhile back I was noticing I was definitely losing emails. As one can might imagine, this is a scary experience since this brings into doubt if the mail system under use is doing something funny to the mail.

My first place to look was in the mail logs for the SMTP server and other associated daemons. However, I saw nothing in the maillogs which was not a very comforting thought.

After more investigation I would notice empty files like this every once in awhile…

~/Maildir)  ls -la new/
total 4
drwx------   2 al  al   512 Jun  8 00:25 .
drwx------  69 al  al  2048 Jun  8 00:25 ..
-rw-------   1 al  al     0 Jun  8 00:22 1244388142.30600_.myserver.net

This gave me more clues on where to look next. So next I looked in my Procmail logs for this particular mail id and noticed the process handling this message was killed by SIGPIPE

procmail: Executing "/usr/local/bin/spamassassin"
[84028] warn: spamassassin: killed by SIGPIPE
procmail: [84026] Tue Apr 14 21:45:26 2009

Googling dug up the following links that explain it all:

 http://www.nabble.com/Zero-exit-code-aft…
 https://issues.apache.org/SpamAssassin/s…

Verdict:
Upgrade Spamassassin

Since I have upgraded Spamassassin, the zero byte email mystery has resolved itself.

Awhile back I installed Ruby 1.9.1 in such a way as to co-exist with my current Ruby installation [1], [2] (I should use rvm [3] these days…)

However, one issue that cropped up during an IRB session was I could not copy and paste Japanese characters into the IRB repl. This is very very painful for my day to day use with Ruby (Imagine not being able to use the ‘|’ character while writing UNIX pipelines).

Below is an example of me trying to enter the character あ into IRB and watching it fail.

$ irb
irb(main):001:0> ab = "?"    <---- Tried entering the character あ
SyntaxError: (irb):1: invalid multibyte char (UTF-8)
(irb):1: unterminated string meets end of file
from /usr/local/bin/irb19:12:in `'

After a lot of head scritching I was able to narrow it down to something with readline:

$ irb --noreadline
irb(main):002:0> ab = "あ"
=> "あ"
irb(main):003:0>

After some more digging into the issue. The root cause seems to be the lack of GNU readline. By default, Ruby will link in the system installed readline library on OS X which is called editline [4]. Unfortunately, editline does not support UTF8 or multi-byte character sets which makes this a no-go for daily usage.

Most of the other references suggest downloading readline from source and installing into /usr/local however I believe this defeats the purpose of using something like MacPorts. After a bit of finagling I found that this is the invocation to get things working.

wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/rub...
tar xvzf ruby-1.9.1-p243.tar.gz
cd ruby-1.9.1-p243
# Don't trust MacPorts version of autoconf because it somehow nuked the
# --with-readline-dir option
/usr/bin/autoconf
./configure --with-readline-dir=/opt/local --enable-shared --program-suffix=19 --enable-pthread
make
sudo make install

I have it wrapped up in a script which you can see here.

References

[1] http://wonko.com/post/how-to-compile-rub…
[2] http://frozenplague.net/2009/01/ruby-191…
[3] http://rvm.beginrescueend.com/
[4] http://thrysoee.dk/editline/

I have a love-hate affair with Capistrano. It is a great tool if you are a Ruby person and need to do something NOW on a bunch of machines. But the docs are in a constant state of suck from my point of view.

The Capify.org website helps for remembering the ’simple’ details on what Capistrano can do. But where I waste a lot of my time is asking questions like, “How do I set the Capistano SSH gateway to a non-standard port?”. Luckily, Capistrano is written in Ruby so it is easy enough to glance through the code and finally find out where it is but this is why good tech docs exist. To give enough context to answer those questions.

To answer my own question, below is a snippet you can add to your capfile to use a non-standard port if you need to deploy through a SSH gateway that lives on a non-standard port

# Add this to your Capfile
# This sets the SSH gateway to a machine called mysshgateway.com on port 22222
set :gateway, ‘mysshgateway.com:22222′

With the announcement of OpenSolaris 2009.06 I thought it would be appropriate to blog a little about a tool I had been writing to help myself play with Zones a bit easier.

My overall goals were the following:

• Have each zone configured with its own virtual NIC (Crossbow)
• Allow easy creation of zones without having to type zonecfg crap over and over again
• Make it a stepping stone to automatically creating zones
• See how well ipkg branded Zones work
• Allow a Zone to get its IP and DNS configuration from DHCP

I had tried going through tutorials that I found on the web (See references below) for setting up Zones but sadly none of them worked to my frustration. After a lot of experimentation I finally pieced together a way to create zones quickly and (almost) automatically for simple configurations.

Howto

1. Create a template zone that will be used as the main clone Zone
2. Download setup-zone-exclusive.sh and modify lines 34-35 to match the name of your template zone and the real interface you want the zones to bind to
3. Download the DHCP event hook script from here and name it dhcp-client-event.sh if you want DHCP configuration
4. Run setup-zone-exclusive with the zonename and the virtual nic interface that you want

In more detail here are the steps below

First create a template zone (I call it barebones here)

# Create /zones as its own ZFS filesystem
$ pfexec zfs create rpool/zones
$ pfexec zfs set mountpoint=/zones rpool/zones
$ pfexec zfs create rpool/zones/barebones
$ pfexec chmod 0700 /zones/barebones
$ pfexec dladm create-vnic -l $REAL_IF vnic0
$ pfexec zonecfg -z barebones
barebones: No such zone configured
Use ‘create’ to begin configuring a new zone.
zonecfg:barebones> create
zonecfg:barebones> set zonepath=/zones/barebones
zonecfg:barebones> set ip-type=exclusive
zonecfg:barebones> add net
zonecfg:barebones:net> set physical=vnic0
zonecfg:barebones:net> end
zonecfg:barebones> exit

$ pfexec zoneadm -z barebones install

Get the script

I would suggest you create a project directory to hold things such as zonecreations.

Download from Github gists here. Name it setup-zone-exclusive.sh. Don’t forget to chmod +x the file so you can execute it

Download the DHCP event hook script

You can get that here. Make sure this script is in the same directory as wherever you saved setup-zone-exclusive.sh

Create a zone

You can now create zones like this:

cd zonecreations
pfexec ./setup-zone-exclusive.sh mycoolnewzone virtualnic1

Have fun!

Update: Fixed an error in the example for using dladm. It should be correct now. Thanks!

References

Downloads

http://gist.github.com/122220 (setup-zone-exclusive.sh)
A DHCP event script to make sure DNS is configured when DHCP acquires an IP

Older docs on setting up Zones on Solaris

How to use sysidcfg file in OpenSolaris 2008.11
Internal Zone Configuration docs
Performing the Initial Zone configuration
Preconfiguring with sysidcfg file
OpenSolaris FAQ on sysidcfg
Ben Rockwood’s blogpost on Zone creation
About /etc/.UNCONFIGURED

Helpful for understanding Zones and Crossbow

Crossbow on vnics

Finding out that there is a change in policy for setting root_password in sysidcfg files

PASSREQ is enforced
zlogin failure after zone setup

The following helped in understanding the role of IPS and ipkg inside a non-global Zone

Updating Zones in OpenSolaris 2008.x
A field guide to Zones in OpenSolaris 2008.05
OpenSolaris forum on sysidcfg and Zones

After looking at /etc/inet/hosts I noticed a loghost entry.

Being a Solaris newbie I was curious to see why this entry was there. A quick Google brought up this nice discussion:
 http://opensolaris.org/jive/thread.jspa?…

Summary, don’t delete it.

On small LAN networks that do not have an internal DNS server. There is a nice technology called ZeroConf that uses multicast to enable name lookup resolution. It has been baked into OS X for quite some time now. Linux and other UNIX flavors have been picking this up as well. OpenSolaris also includes this but enabling it is not on by default (At least with 2008.11). Here is a quick howto.

Edit the file /etc/nsswitch.conf and make sure that the line that begins with

hosts:

contains the following

hosts: files dns mdns

Then you should be able to ping any machine that uses Bonjour. For example, if you have a Mac that is named mycoolmac then you should be able to ping mycoolmac.local

References

• Heads-up Multicast DNS and discovery

Tom has a great summary on why.

One challenge for the cowboy sys admin is on how to keep oneself engaged while making their job basically… a walk in the park.

One thing I have found helpful in creating lists is to be dogmatic about writing docs as you are doing something somewhere, anywhere and collect all of this later. (You are writing documentation as you do your job, aren’t you?)

Read More

After reading Game Developers and Porn Stars I started recollecting an earlier time in my life. At that point in time I was considering a life as a game developer. I had heard the rumors that life as a game developer was a meat grinder and had really long hours. I spent time reflecting on the choice I had. I really like video games and think they a great form of entertainment that has had a large influence on my life. But I still feel, at its core, video games are just entertainment. Sometimes, they can educate along with delight but that is all.

Ultimately, I took a different path than becoming a game developer. After reading Kill Ten Rat’s blog post on Game developers I am glad about my choices. I have pretty much erased almost any regrets on not taking that path in life. Although I AM sad to read such a story in 2009 because the decisions I made were over a decade ago and it is disappointing to hear the state of the game industry for a game programmer as a whole seems so soul crushing.

Seems someone else ran into issues while trying to deploy god.

While, I don’t think god sucks I definitely don’t endorse it. At this point I would only use it under the following conditions:

• Need for a process monitor tool with more dynamic configuration setups. This is where god really shines against monit’s simpler understanding of what process management is about.
• The host that needs monitoring can easily spare at least 16MB for a monitoring process. See below on why.
• I really want an all Ruby solution for all the tools in a system

In general, I am into the whole ‘It is Open Source. If you’re having issues, fix it’ deal so I am not nearly as angry sounding as Brad is about god. However, after having issues with god, I switched to monit for simple process monitoring and restarting. I had far less troubles and got on with other tasks that I considered more important than perfection in a process monitoring system.

For those that are curious here are the issues that I ran into with god:

• Daemonized Ruby took at least 8MB of RAM for the monitoring process. With RAM the way it is, this is not as big a deal. However, if you are trying to get by on a 128MB VPS host every kilobyte counts.
• God itself had issues just randomly dying after some time. Tom promptly fixed it after it was reported and that was great. However, it was a little disappointing that a monitoring process just died.
• Sparse documentation compared to monit’s. Then again this is typical from many Ruby projects and luckily Ruby code is readable enough
• Digging up known issues for god required noodling through groups, forums, and blog posts. Would have been nice to just have a friggin’ FAQ like other sys admin-targeted software I have seen.

I also DO agree as has been said in the comments on Brad’s post that it is the responsibility of the deployer of software to handle the issues with whatever they deploy and just deal with it. The reason I say this is because I fell for the hyped up description of god in the beginning and ultimately paid the price when it sucked up my time. I dealt with it but definitely am less impressed with overhyped marketing descriptions of software these days. Personally, I am not a fan of that type of marketing for software since it seems a little disingenuous to me. But that is just me.

I had a need for inserting the name of the current file into a bunch of files I was editing. I was pretty sure there was a function to do this in vim and after some searching I was right.

To insert the current filename. In Insert Mode, type CTRL-r % and it will insert the current filename.

Thanks to blog post for the tip!

A nice back story on Writely (what is now known as Google Docs) via HN. I particularly like this snippet

[The creators] have been in the application software business for nearly 20 years… they understand the user problem so deeply that they can blend the advantages of each new platform with ‘document authoring problem’ to really build a platform-native solution, not a clone of someone else’s work.

Read more

Akihabara as many people in Japan know was originally the home for buying electronic goods in the Tokyo area. It still holds that reputation however the Anime Otaku crowd have changed the face of Akihabara to also accomodate their needs and desires.

One thing that is interesting is the origin of place name Akihabara. A friend of mine has an excellent post here

Read more!

After having dain bramaged myself for years with Linux usage. I had gotten spoiled into believing an OS should make it simple to do the following:

1. Shutdown computer
2. Swap around hardware components
3. Restart
4. Life is good

However any techie should tell you this is a pipe dream on Windows. Mac users probably have no clue since they never change hardware components and just buy new Macs to solve their problems. Which leaves the lucky OSS *nix variants to try stunts like this.

Being the stubborn person I am, I attempted this with OpenSolaris by swapping out my motherboard. I wanted to do this in order to take advantage of the E7400 Core 2 Duo that I bought awhile back. Things almost worked however on reboot I was given the dreaded console login screen with a useless keyboard. The following as far as I know don´t work…

1. CTRL-ALT-BACKSPACE
2. CTRL-ALT-Fn
3. CTRL-ALT-DEL

Your best bet is to ssh somehow and try to look for clues. Here is what I did…

1. Swap motherboard and stare at dark screen
2. Find out how to boot into single user mode and make sure the kernel isn’t PO-ed or something and find my IP address
3. Move away the X11 configuration that I configured (dual-display) and try rebooting
4. Reboot and find out it isn’t working
5. ssh in and realize it still isn’t working. Move the old dual display X11 config back to /etc/X11/xorg.conf
6. Try restarting gdm with svcadm restart gdm and watch it fail
7. Scritch head some more
8. Try starting X from the SSH session and whoah it works
9. Restart gdm (svcadm restart gdm) and now I get a login screen
10. Realize that I disconnected the left monitor (VGA) to help debug and want it back
11. Logout and log back in. I now have dual screens and a working Solaris install again!

References

• How to boot into Single User Mode for OpenSolaris
• Finding device faults with Fault Manager
• Debugging problems with nvidia driver on bootup

I’ve been silently scanning some Solaris blogs and skimming some of the appropriate websites for documentation on some of the more interesting features of Solaris for awhile now. One thing that requires time to adjust to is the number of acronyms that the Solaris community has to describe their technologies. One of these is called L2ARC.

At first I thought it was some hardware device however after a bit more searching it turns out that it is part of the ZFS technology suite. L2ARC stands for second level ARC where ARC is a read cache system for ZFS that uses a system´s main memory for holding the cache. While ARC uses the hardware´s main memory, L2ARC is designed to take advantage of faster I/O media such as SSD devices to provide faster read throughput than what a typical hard drive can offer.

Brendan Gregg has an excellent overview explaining what the L2ARC is and some of the benefits it can give in accelerating random reads.

According to the Fedora FAQ one should be able to update with just this

sudo yum --enablerepo=updates-testing groupupdate "KDE (K Desktop Environment)"

However when I did, I ran into some icons from packages kdepim-3.5.9 and kdegraphics-4.1.0 conflicting with packages crystalsvg-icon-theme and libkipi. Here is a log…

  file /usr/share/icons/crystalsvg/48x48/apps/kpalmdoc.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_contacts.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_date.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_journal.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_mail.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_news.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_notes.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_summary.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_summary_green.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/kontact_todo.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/crystalsvg/64x64/actions/rss_tag.png from install of kdepim-3.5.9-10.fc9.i386 conflicts with file from package crystalsvg-icon-theme-4.0.4-1.fc9.i386
  file /usr/share/icons/hicolor/16x16/apps/kipi.png from install of kdegraphics-4.1.0-3.fc9.i386 conflicts with file from package libkipi-0.1.5-4.fc9.i386
  file /usr/share/icons/hicolor/22x22/apps/kipi.png from install of kdegraphics-4.1.0-3.fc9.i386 conflicts with file from package libkipi-0.1.5-4.fc9.i386
  file /usr/share/icons/hicolor/32x32/apps/kipi.png from install of kdegraphics-4.1.0-3.fc9.i386 conflicts with file from package libkipi-0.1.5-4.fc9.i386
  file /usr/share/icons/hicolor/48x48/apps/kipi.png from install of kdegraphics-4.1.0-3.fc9.i386 conflicts with file from package libkipi-0.1.5-4.fc9.i386

Error Summary
-------------

I tried deleting these packages manually but that led into an even deeper’s rats nest of dependency hell. So one thing I tried was

$ sudo yum groupremove "KDE (K Desktop Environment)"
$ sudo yum --enablerepo=updates-testing groupinstall  "KDE (K Desktop Environment)"

However I still got conflcts so ran

$ sudo yum remove libkipi
$ sudo yum remove crystalsvg-icon-theme
$ sudo yum --enablerepo=updates-testing groupupdate  "KDE (K Desktop Environment)"

And finally I have got KDE 4.1.0 to install in way too many steps.