You’re clearly not doing anything for which VMWare is a help. Therefore, you are (as you clearly suspect) just wasting computrons on it, and adding unnecessary complexity as well.

VMWare is great if you have a proper computer on which you need to run a couple of applications that only work with a garbage OS. (I’m using it because my fiancee is dependent on a Windoze application. I run VMWare on one of my servers, with TightVNC running inside the virtual machine, and she can connect to it and do her thing whenever she needs to. And when the Windoze instance gets gorked, as they all inevitably do eventually, I can just restore a snapshot I took shortly after I finished the configuration, and everything is fine again.)

VMWare is less great than that (but still very much a viable option worth considering) if you’re virtualizing a large distributed application in such a way as to provide for more dynamic resource allocation and more effective redundancy. If you’re devoting a meaningful fraction of a data center to your application, and wondering about whether you might be able to structure it more efficiently by using virtualization, the answer is probably “yes”, and VMWare can be a good part of the response.

But if you’re only throwing one physical server at the problem, and the problem itself is one that is implemented entirely in a modern unix, there’s absolutely no benefit to VMWare that can’t more easily be achieved by a lighter-weight solution that doesn’t require heavyweight resource partitioning and virtualization overhead. If you think it can be done with standard unix filesystem tools, it probably can, and that’s probably the best way. If you think it can’t, you’re probably wrong. And if you KNOW that you can’t, you’re probably still better off with something like UML under Linux or jail under FreeBSD.

I would also check for disk contention. First use a tool like bonnie++ to see what the array can handle for disk IO. Then, when under a load of all 4 VM’s, run iostat to see if you’ve hit the practical limit for random reads/writes.
That will let you know… if IO is the issue, you either need fewer VM’s or faster storage (e.g. more spindles)

The thing that i still do not understand is why and when the problems started. I am using these virtual machines for 3 or 4 years now and suddenly (maybe 6 month ago) these problems started. I just can not figure out when and why.
Maybe it was the switch to a tick-less kernel ?

For more background on the changes that I think were key to getting my system running well, see

http://vmfaq.com/?View=entry&EntryID=25

and

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=844

http://www.xen.org/

PS – Turning off the software RAID cut the time needed to execute a 32,000 row update in half.

I disabled cpuspeed on the physical host and now the VMs appear to run their clocks at the proper speed. The issue that remains is that they appear to be getting the wrong time on bootup. The time they do display seems to be fairly random but generally a few hours and minutes into the future. I thought about perhaps installing NTP in the VMs as well to get a good time on boot and then let VMware tools takeover from there but if NTP sets the time even a second ahead of what VMware tools sees on the host the time will not be synced by VMware. I suppose this would only be an issue if the clock ran fast. Do you guys think it’s safe to assume that the clock will no longer run too quickly and if it runs slow VMware tools will pick up the slack?

Moving onto the issue of the harddrive problems. I ran diagnostic checks on the drives themselves and they passed just fine. Intrigued by the suggestion that this was due to software RAID I broke down the RAID and booted the server from one of the drives. The log files no longer contain the error message and as far as I can tell the system has stopped freezing for seconds at a time. I think it’s now time to evaluate the situation and decide what to do (we’ll have to decide whether we can live without RAID, whether we want to move to hardware RAID, or whether despite figuring out this problem we’d rather wipe everything so that we can use software RAID). Regardless of the direction we take I think this page serves as a testament to the fact that VMware Server and software RAID on Linux do not play well together. Let this be a warning to others thinking of such a configuration.

If this is for research/educational use, you can get ESX standard (two physical CPU) for free, I’d recommend it over Linux/Win + VMWare Server.
http://www.vmware.com/partners/academic/