LOAD BALANCERS:
My understanding is that Willy Tarreau’s HAProxy is the premier open-source HTTP load balancer today. I have not used it myself, but it has a good reputation and seems well designed (http://haproxy.1wt.eu/). It’s author has written a great article, which gives a solid overview of best practices for load balancers today, at http://1wt.eu/articles/2006_lb/ .

I have used Pound myself, and it’s really nice (www.apsis.ch/pound/). Strong points are SSL support (not a requirement here) and ease of setup. I’m not sure if Pound is the optimal choice for a high-performance scenario, but then again I really don’t know how fast Pound can go. Consider hanging back in the older version numbers of Pound, sometimes the newest versions have less well tested functionality.

I have never used Danga’s PerlBal, but from what else I have seen from those guys I would have no reservations. Their site has a very good presentation about high-performance, dynamic-content web architectures  www.danga.com).

The new kid on the block will probably be Varnish Cache. It’s brand new and not quite polished, but it’s architecture will allow it to scale beyond all the other contenders, and I know it’s main programmer to be really sharp. It’s probably not ready for your use now, but it should grow within the next year  www.varnish-cache.org; http://www.nuug.no/aktiviteter/20060919-varnish/)

Looking towards the future, it seems to me that the trend is to move away from the classic HTTP load balancer. Today HTTP servers have load-balancing built in, and can serve static content directly from the front-end server (the “load-balancer”), while balancing retrieval of dynamic content from back-end app servers via HTTP. Apache 2.2 can do this out of the box, I believe LightTPd can too  www.lighttpd.net), and I know that Litespeed Web Server 2.2 can  www.litespeedtech.com). The Litespeed server has a really good rep in the Ruby On Rails community; Lighttpd has a strong rep for high speed but has been a bit brittle in use. The website of Mongrel HTTPd has good docs on configuring these servers (http://mongrel.rubyforge.org > Documentation).

Performance is really a non-issue of sorts today. If your preferred load balancer / front end HTTPd isn’t fast enough, then you just add more copies of it. Typically you’d go with two cloned LB’s, DNS round-robin between the two, and IP level fail-over. This setup is cheap, and gives you good protection in the event of traffic spikes OR hardware failure (but not both a traffic spike AND a LB failure). There are many IP level fail-over systems, Linux HA, Linux Virtual Servers VRRPd etc., but Whack-a-Mole seems to be a favorite despite it’s age  www.backhand.org).
Another thing is that your usage is atypical. Most LB’s hit their bottleneck in the session creation phase, ie. how many new TCP/HTTP sessions can be established pr. second. Photo.net would create most of it’s load by sending large files in long-running TCP sessions. The internal design and buffering strategy of the balancer would matter much more than usual. I can’t say how this will work out, but my guess is that 150 Mbps in long-running TCP sessions is a piece of cake for a single contemporary server. Benchmarking your specific setup is really the only way to know.

APPLIANCES:
Loads of companies make load-balancing switches, but the prices are all over the map. Radware, Coyote Point, Zeus, Cisco and Juniper are some of the names. Perhaps some of these can be found at reasonable prices in your area, perhaps not. Firewalls are more commonly purchased as appliances, partially for ease of setup and layering of security responsibilities across multiple systems for resilience.

HTTP PERFORMANCE:
There is a small end-user performance benefit to spreading the HTML/CSS/image content over multiple host names, such as http://www.photo.net and static.photo.net, because the end-users browser then opens more parallel TCP connections. See http://yuiblog.com/blog/2006/11/28/performance-research-part-1/ for an introduction. I do not think this is worth optimizing for today, and would personally ignore it for now, but it is real.

MY CONCLUSION:
Load balancing is cheap and attainable today, also at 150 Mbps. The preferred setup really comes down to individual taste and past experiences. Personally, for a new site with those requirements, I’d first try Litespeed HTTPd, and if benchmarks show insufficient performance, then I’d add more HTTP front ends with Whack-a-Mole IP fail over. HAProxy on a dedicated load balancer PC also seems like a good, problem-free approach. Firewalling I would try to keep on a dedicated appliance, mostly so that I don’t accidentally create security holes while mucking around with IP settings on the load balancers.

Anything with 2x gigabit NICs (they have better chipsets than 100mbit NICs) and a P3 processor will be able to handle your load quite easily.

For #2, :

Get a P4 machine for better SSL encrypt/decrypt, and make the proxy machine handle SSL connections as well. Then you can set a header value to let AOLserver know that the connection was SSL-secured.

For reverse proxying, you can use either Pound (www.apsis.ch/pound) or Apache with the mod_proxy and possibly mod_rewrite module.

Given the massive traffic photo.net sees, I doubt that a Perl based program would be the best solution. Everything I have mentioned is open-source.

Some of the stuff could be done inside a switch, but it would be more expensive and not opensource.

It is possible to engineer it so you have two switches and each server has an uplink to each switch. This protects you against a switch failure but introduces additional complexity and things that can go wrong. Unless you have a network geek around, this could end up causing more downtime than would have resulted from a single switch failure anyway.

I have used LEAF (Linux Embedded Application Firewall – http://leaf.sourceforge.net) for border routers and firewalls with excellent results. It comes with Shorewall which makes complex iptables configurations easy and Quagga in case you need to do BGP. There is a module for keepalived which I’m pretty sure would allow you to load-balance your webservers.

Fazal: I looked at that Cisco load balancer. It isn’t clear that patterns of the kind that I’m talking about can be implemented. What’s worse, it doesn’t look as though these things provide any firewall action. So we would be introducing an extra point of failure.

The first problem is an IP Firewall. There’s really very little to say here. iptables comes with linux and works remarkably well. Mix this with the right network architecture, and you’re in business.

The second problem is persistent load balancing. You want to be able to define services based on protocol (port 80/443 to the web servers, 25 to the mail servers, etc, etc) And, preferably, if a user connects to web server #3, and then makes another request within X number of seconds their second request is also routed to web server #3. The LVS project actually manages this quite well (and cheaply) see: http://www.linuxvirtualserver.org including persistence (or affinity in cisco-ese) LVS also supports a couple of different connect and passthrough models for different requirements.

The third problem is that of load balancing your content (Thumbnails on web cluster 2 versus original photos on web cluster 1 for example) Your request was to be able to do that via the URL. This is where the answer gets a bit more ambiguous. You can use a reverse proxying setup (perlbal and pound for example) if you’re still interested in a URL based forwarding pattern. But if you were willing to dedicate an IP address you could use DNS based load balancing. Take the following setup, for example.

Request for server.com comes into the load balancer server.com is a.b.c.d. The load balancer knows that the web1 web2 web3 and web4 servers on the LAN are able to handle web requests for the ip address a.b.c.d. The load balancer hands the request off to web2. Further requests from the same client ip address in the next 180 seconds (because of persistence) also get directed to web2. A thumbnail appears on the page at thumbs.server.com which has a different ip address of a.b.c.e. The load balancer knows that web5 and web6 are set to be the real servers for a.b.c.e and the request is sent off to web6. Because persistence was not configured for a.b.c.e like it was for a.b.c.d the next request might be sent to web5 since all requests are round-robin load balanced for that IP address. Any time a web server goes down the mon (or other monitoring daemon) running on the load balancer notices and takes it out of rotation. and future requests are routed around the down web server until it is brought back up. Heartbeat detects if one of the LVS load balancers goes down and has the other assume its position.

In the above scenario you don’t really get to use the URL as part of the balancing mechanism, but you do get to direct based on content type.

You could also insert into the chain a reverse proxying solution (or cluster) and have the best of all available worlds by load balancing, in a highly available fashion, to the proxy servers

(Request) [ 2 LVS machines (public IPs) ] [ 4 Reverse Proxies (priv) ] [ X "real" web servers (priv) ]

It’s not as elegant as a set of $50k hardware load balancers, I’ll grant you, but it’s workable and a LOT cheaper. And I would definitely recommend not looking for an all-in-one solution for the problem in OSS (if you are) and start thinking in terms of chaining the proper functionality together.