Brave New Virtual World

A friend has a lightly used Magento-based ecommerce site that delivers pages more slowly than he would like. He reported that there were only about 100,000 page views per day, i.e., an average of about one per second. I said “Are you sure that you have enough RAM to keep the database in memory?” This was not an issue that his team of developers had considered. After some digging I found out that he was paying for two servers, each with 32 GB of RAM. On these two physical machines he was running three virtual machines, one for the MySQL database management system, one for the Web server, and one for a staging server. These VMs were allocated 16 GB of RAM for each production VM and 4 GB for the staging server. In other words, he was paying for 64 GB of physical RAM but the Gods of virtualization had set things up so that his production site could not possibly use more than 32 GB of RAM. The database VM ran on a physical machine by itself, leaving 16 GB of RAM entirely idle except for whatever the hypervisor was using.

Requests for the site and server documentation were answered with “What’s documentation?” (I pointed to “Software Design Review” for examples, but still not a single document was found for a site in which at least several developer-years have been invested.)

Does virtualization really make sense for people who aren’t going to do basic design work or write down simple documents that say (1) how much data there is, (2) where the data will be stored, (3) how much memory will be required at each point where data are stored? This particular site would almost surely have run better if the people who set up MySQL had never heard of VMware.

5 Comments

  1. Fazal Majid

    April 25, 2014 @ 2:07 am

    1

    It is hardly fair to blame virtualization for what seems like a clear case of incompetent programmers with no clue on systems administration or performance tuning. They would have delivered just as dismal a result on bare metal.

    A simple e-commerce site should not need that much RAM to run efficiently in the first place. I suspect Magento, like WordPress, is quite bloated with layers and layers of abstractions and hooks for customization. Caching modules like APC or the newer opcache should help with that. More likely, the database is setup incorrectly (e.g. using MyISAM instead of InnoDB as the storage backend), and something like the Xdebug profiler would help identify the bottlenecks.

    There are a number of consultancies like Percona, SkySQL or OmniTi that would be happy to help, and from what you are describing there should be low-hanging optimizations that could be implemented quickly.

  2. michiel

    April 25, 2014 @ 2:54 am

    2

    Virtualization offers different benefits to different groups of people. For companies with large data centers, it’s about overcommitting underused hardware. For system administrators, it seems to be about agility (they can spin up and retire servers without waiting for new hardware, or even having to go to the data center) and configuration management (it allows them to configure many small single-purpose machines that are easily replicated many times over). Some people see it as a good way to mitigate security issues by running software in isolation.

    From the perspective of someone designing a static system, virtualization is often a sub-optimal choice, and it certainly is in the scenario you describe, but there seem to be serious benefits for people who work with systems that are dynamic, and not ‘designed’ per se.

  3. Joshua Levinson

    April 25, 2014 @ 9:35 am

    3

    I will say the same thing I said to you when we were discussing Rails. Any technology is harmful in the hands of the unskilled. The world’s best Java developer and Database designer, might still produce the world’s worst JavaScript and MongoDB based application.

    As a corollary to this, the more complicated and the newer the technology, the greater mastery that is needed to avoid doing harm. Java and MySQL have been around long enough that it takes less mastery to do harm with them. You might not write the most modern or world’s most elegant solution with them alone, but the odds of you doing something harmful to yourself or your customer / client is lower.

    Virtualization certainly isn’t new, but it is relatively complicated, and even very experienced software developers may never have ever had to think about the hardware their code runs on, especially if their experience is at larger organizations that separate infrastructure and operations from development.

    This may be unrelated, but it’s also worth pointing out that a contract engineer may still do harm by using a technology that they have mastered, if he/she knows that the customer who the solution will be delivered to does not have the same level of mastery, and would be unable to support that solution without the original developer.

  4. Baz

    April 27, 2014 @ 12:00 am

    4

    I would be surprised if more RAM would help. Most Magento databases are around 1-2GB w/ all indexes etc. Magento is usually CPU bound on the app server layer.

    Step 1, install New Relic and see how much time is actually being spent in the database. Step 2, utilize Varnish w/ a Magento extension like Turpentine to utilize ESI and reduce the work Magento is doing.

    Because Magento is a terrible beast and the trick to getting it to perform is to have it do the least possible.

  5. jack crossfire

    April 27, 2014 @ 8:27 pm

    5

    Any good computer science project uses multiple virtual machines, 15 languages, & some cloud services.

Log in