It looks like one of your former SEIA students, Dan Chak, just published a book on scaling Rails.

It’s called “Enterprise Rails,” and Hal Abelson reviewed it, saying, “Enterprise Rails is indispensable for anyone planning to build enterprise web services. It’s one thing to get your service off the ground with a framework like Rails, but quite another to construct a system that will hold up at enterprise scale. The secret is to make good architectural choices from the beginning. Chak shows you how to make those choices. Ignore his advice at your peril.”

http://www.amazon.com/Enterprise-Rails-Dan-Chak/dp/0596515200/

Chak comes from an ACS/AOLserver background and worked at Amazon, and I started using ACS/AOLserver back in the nineties so I was intrigued by his view on Rails.

All the best.

James

First off I believe in giving MySQL all the memory I can, the more memory the happier it will be, period. Memory is so cheap these days, not doing so makes no sense. It is preferable to limit the dataset size to fit in memory but that is not always possible, but at least make sure you can fit the indices in memory. Also it is best to allocate the memory to MySQL as opposed to the file cache for Linux. Bear in mind that there is overhead to MySQL, so for InnoDB a good rule of thumb is 3x overhead.

Also there is some patterns to avoid when you are trying to get the best performance out of MySQL such as avoiding joins, large selects, table scans, and unindexed selects, updates and deletes. Nothing out of the ordinary really.

I am usually against using Memcached for a few reasons. Why allocate memory to Memcached when you can allocate it to your DBMS. Using Memcached adds another level of management to the application which adds to the complexity (aging data, etc…) Memcached is usually trotted out to work around performance problems rather than fixing them because it is a quicker solution. Finally Memcached stores blobs of data which need to be serialized and deserialized. That being said, I would use Memcached where it makes sense.

I was not aware that Linux limited the amount of space allocated to caching files, I think that may have changed, my understanding is that current releases are very aggressive with caching, but I could be wrong.

Running Linux with no swap is fine, I know of some MySQL admin who do it. The only thing to be aware of is that Linux will kill off the largest user process if it runs out of memory, which on a machine running MySQL is usually MySQL.

One trick I have used in the past it to allocate as much shared memory as there is RAM in a machine and to copy files from the file system to the /dev/shm file system which linux tries to keep in RAM as much as possible. I wrote a post about this a while back: http://fschiettecatte.wordpress.com/2007/05/27/sharing-the-memories/

As for RAM utilization, your data set may fit in 2.5GB today, and make an application that makes full table scans (due to poor coding or lack of indexes) look like it is performing acceptably. But once the data size hits 2.500001GB, performance is going to collapse. It would be worth doing some profiling to find out which pages cause the most problems. Obviously, if the site is failing and time is critical, upgrading RAM is a sensible band-aid, but you can’t stop there.

Regarding the Linux buffer cache problems, Linux’s VM subsystem has been unnecessarily rewritten so many times between 2.4 and 2.6 it’s not surprising even experienced sys admins could not figure it out. Solution: use a non-toy OS like Solaris or FreeBSD.

Finally, regarding Microsoft solutions – I have plenty of experience with Fortune 500 corporations who have Windows web hosting farms where IIS can’t even handle static file serving without clogging up with memory leaks, and our best Windows experts have absolutely no tools to figure out what the hell is going on. Solution: use the F5 load-balancers in front of the Windows farm to offload the said file serving to Apache running on Solaris machines with half the hardware resources. ASP.NET is fairly efficient, far more so than Java, but it is subject to sudden degradation in performance and once that happens you are in a hole it is very difficult to dig out of.

I personally appreciate your caveman approaches. Your writings over the years about embracing simplicity in software and system design have been valuable.

It is extremely important to know what is going on under the hood of anything you do in code; the MIT Genius seems to have forgotten that. The virtues of the caveman approaches are that you know exactly what’s going: this query is being run, and over here I loop over the rows, fetch it into RAM, or whatever.

The danger, in my view, is that modern languages have first paved over memory-management concerns (garbage collection) and now database overhead (ORMs). Lo and behold, we now have a new swath of powerful tools, all ready to shoot ourselves in the foot. Your ASP users were fortunate that they had no such tools at their disposal.

The lesson seems to have been forgotten: adding abstractions to a system generally makes it harder to understand and fix performance problems, and often makes them easier to create. The defensiveness of the rails developers on this point is bizarre.

Great post. Some observations:

1) Programmers think modern “abstractions” protect them from having to worry about stuff like SQL, hardware, etc. Joel Spolsky of JoelonSoftware.com covered this very well in “Law of Leaky Abstractions” (http://www.joelonsoftware.com/articles/LeakyAbstractions.html)

2) At most large shops the “database guy” and “developers” are two different classes of citizen. “Developers” are “real programmers” whereas database guys…or database guys. The two don’t overlap. “Real programmers” don’t worry themselves with the database.

One of the alternatives to ActiveRecord is Datamapper (mostly associated with Merb).

You’ll find more details here:

http://datamapper.org/doku.php?id=why_datamapper

With Rails 3.0 coming up, DataMapper may well get some more attention.

rgrds,

Johan

I’m a big fan of Ruby on Rails and I’m really surprised by the negative reactions from the community. It seems to me your article aligns very well with the philosophy of RoR: keep it simple.