Sunny Days – Day 9

I know, I promised some NFS benchmarks. But due to heavy workload and some recent events, I didn’t quite make it. But I did manage to run some other benchmarks that may be of interest to you. Since I had Apache up and running the obvious thing to do was to run some benchmarks to measure the web performance of the Sun Fire T2000. But then, running a performance benchmark against a web server isn’t that easy, is it? There’s a bunch of tools out there to do that. Most of them have one major drawback: they’re not only commercial but they’re also expensive. But my intention was to run a test that could be run by everyone at little or no cost so that everyone would be able to compare my results with his or her system. So here’s what I came up with. Feel free to comment and tell me whether I did something wrong, whether you’ve got another idea about how to test or any other things that come to your mind. If time permits I’ll see what I can do to run other tests *you* suggest. But now, without further ado: the benchmarks and the software and the environment I used to run them:

The Software
Since the majority of web pages is served by Apache and the T2000 won’t run Windows anyway, I decided to run an Apache based test. So why not use the Apache Benchmark which is included in the apache package and therefor should be available almost everywhere where Apache is running? For this seemed to be a good solution I went with it.

A harder problem to solve was to find a generic test scenario that would be easy to reproduce. My intention was to run test against static pages and pages that were dynamically created using PHP. The static part was easy to solve: I’d just use the demo pages included with the Apache server. They’re the same on every platform so there would be the same amount of data to transfer. To eliminate overhead caused by the automatic language selection I adressed index.html.en directly. But what about dynamic pages?

Some would suggest to use a simple call to phpinfo(). To be honest, that was my first idea, too. However, I found that the size of the page generated by phpinfo() varies a lot – even if you compile your own PHP interpreter. The difference in sizes I found was around 10 to 20 KByte which is more than significant enough to make it useless for a fair test.

Thus I settled for a base install of Textpattern, a small, yet very flexible CMS or — depending on your point of view — weblog system. Using Textpattern not only gave me a page with an almost fixed size (the time displayed would change so there was a difference of some bytes I found to be negligible). For Textpattern stores the content in a MySQL database and creates every page on the fly, the results of the benchmarks should also give a good idea of how well the AMP components (Apache – MySQL – PHP) would work together on the systems. Also, Textpattern can be set up in under five minutes so it’s no great deal to install the software and run some tests on your system.

The Systems
Obviously, the next question was what machines to match with the Sun. I went for mixed-pickels ;) So the first machine in the test was to be my trusted Compaq ProLiant DL380 G1. Sporting two Pentium-III CPU’s clocked at 733 MHz and accompanied by a total amount of 1024 MByte RAM that’s clearly no match for the Sun. But it’s what some of my customers still use. It’s a trusty old work horse and I just wanted to see how it would stand up against modern technology. The OS on the ProLiant was OpenSUSE 10.0 with all the latest patches applied. PHP was compiled to match the setup on the Sun, Apache and MySQL were the ones from the Linux Distribution used.

The second system was a Lynx Workgroup Server 4200. Equipped with two Dual-Core Xeons clocked at 3,0 GHz and 2 GByte of RAM, it’s got more horsepower than what you’ll probably get when renting a server at any given ISP. On this machine the OS used was SUSE Linux Enterprise Server 9 (64-Bit edition), latest patches applied. Also, I used PHP and MySQL as included in the distribution. So there may have been an advantage there for the standard version has got a smaller footprint that what I normally use. I tried to compile my own but found this to be difficult because of the mix of 64- and 32-bit libraries on the system.

Lastly, I used the Sun Fire T2000, standard edition with 6 cores and 8 GB of RAM, also with the latest patches to the OS applied. The Apache web server used was that came with the machine, MySQL was version 5 from BlastWave, PHP was compiled as described in Sunny Days 5.

All systems were connected to my 24-port Netgear GS524T Gigabit Switch via Gigabit Ethernet. Although the Sun sports four and the Lynx has got two gigabit ports, only one port was used in the test. One could argue that this wouldn’t allow for load balancing and that’s true. However, you’ll rarely find a server that’s connected to the Internet using a gigabit pipe, let alone multiple ones. Also I wanted to know whether I’d be able to saturate the pipe.

The Test
My client to run the tests from was a generic Pentium-IV machine, clocked at 2,66 GHz with 1024 MB of RAM and a gigabit ethernet connection to the switch. It ran under OpenSUSE 10.0 with the latest patches applied. The Apache Benchmark used was the one shipped with the OS.

The test itself consisted of multiple runs where 1000, 2500, 5000, 7500 and 10.000 requests for a static page (/index.html.en) and a dynamic page (/tp/index.php) where fired at the server. The tests were run in two flavors: Once without using the keep-alive feature of modern web servers and browsers and once with keep-alive enabled. The keep-alive feature allows for multiple transfers within one session whereas the normal behavior would be to open a separate session for each transfer. So clearly one would expect to see better results with keep-alive being active.

In addition, each of these 20 tests (five different numbers of requests to static and dynamic pages with keep-alive enabled or disabled) was run with an increasing number of concurrent connections from the client. The numbers of connections used were 1, 10, 25, 50, 75, 100, 250, 500, 750 and 1000. This was done to simulate an increasing number of simultaneous requests on the target.

To reduce the overhead on the web server’s side, all requests were done using the IP address of the server in question instead of the hostname. Since the requests were made for the same page over and over, there were no directory traversals and close to no file lookups. Basically this eliminates all factors that could have a negative impact on the raw performance. What can’t be eliminated is the loading of the PHP interpreter so I had to trust on the caching abilities of the operating systems to help out there. So, to sum it up, this isn’t a real world scenario but instead gives a measure of the raw performance. I’ll try to hack up a benchmark with directory traversals or even emulating a large bulletin board later, if time permits. You can find a link to the archive with the scripts I used for this benchmark at the bottom of this page.

The Results
Well, before I get into the details, let me again state that this was no real world test and that the scenario used was tuned to measure raw performance only. So you have to take the results with a grain of salt. Still, they are very interesting.

Executive Summary: If you’re running a web site, get rid of any old hardware you might still own. The Proliant clearly was no match for the other two. No (big) surprises there, I’d say. If your web sites use static pages only, you may want a Sun Fire T2000 to serve them. The overall performance is better than what the really well equipped Lynx server provided. Still there’s the money issue. You can buy two Lynx for one Sun. That however is only upfront cost. We also have to look at power consumption. I am still waiting for my ampere meter, but my initial guess is that the Lynx will consume considerably more power and thus make it more expensive to run. Lastly, if you’re using dynamically created pages using PHP and MySQL you definitely want a Sun to serve them. Not only because the Sun came out first in all the dynamic tests but also because it’s got a much more predictable pattern of serving the pages than the other machines.

An Aside: Sun claims that the Sun Fire T2000 is a silent server and it is. Make no mistake: It’s by no means noiseless. When turned on, it will create a noise level that I’d call more than acceptable for a server. Working in the server room when the Sun was running was no problem for me. Now, the same can’t be said for the Lynx Workgroup Server 4200. In addition to the high-level hardware it’s got something I dubbed “Jet Scream Engine”. When turned on, the noise emanating from the machine is unbearable. I had to wear ear protection to be able to work in the same room, even when the machine was stored away in the enclosure. So would I say the Sun Fire T2000 is a “silent server”? You bet.

As you can see, when handling 1000 requests, both the Sun and the Lynx are slow to react when the number of concurrent connections is low. Maybe that’s a “come back later when there’s some real work to do” attitude ;) The Proliant starts off quite good but stays at around 2000 requests per second during the whole tests. The Lynx takes a rocket start at 10 concurrent connections but drops off fast. The Sun starts almost like the Lynx but doesn’t drop off but instead increases speed. Why it then drops at 750 and 1000 concurrent connections I can’t say. Must be some kind of limit reached there, for it’s visible in all the tests with static pages and keep-alive not used. I’d be happy if someone knowledgeable with the Sun could give me some hint there. My best guess is that we’re hitting the server at a rate faster than new Apache servers can be spawned. So tweaking the number of pre-spawned servers might fix this. I’m going to try this shortly.

Serving 2500 requests the Sun shows a similar pattern as in the previous run. However, the drop in requests per second this time starts at 250 concurrent connections. The Lynx is hit even worse by the same phenomenon and the Proliant’s rate of pages served drops considerably too.

Same pattern with the Sun again showing a graph that’s a lot more predictable then the others.

The most interesting thing when running 7500 requests against the server is that now the Sun has the upper hand when doing 750 and 1000 concurrent connections.

At 10000 requests the Sun shows a clear advantage in the medium range of concurrent connections and is about as good as the Lynx at high rates of concurrent connections.

The picture changes a bit when using the keep-alive feature:

We still see the “slow” start from the Sun, but it won’t drop that dramatically when the number of concurrent connections increases. Similarly, the Lynx performs better but loses at the end. Still, the numbers of pages served compared to the run without keep-alive being active is somewhat stunning.

Nothing changed in the start-up behaviour when doing 2500 requests. The Lynx, however, falls back very fast and doesn’t really recover whereas the Sun is doing pretty good once it has gained it’s peak level at 25 concurrent connections.

At 5000 requests the steady behaviour of the Sun becomes even more apparent. The Lynx again starts faster but drops even further down to then recover at 250 concurrent connections. But in the end the Sun handles a lot more requests than the Lynx and the Proliant combined.

When handling 7500 requests, the Sun acts quite unimpressed and keeps going strong after the by now well known slow start. The Lynx this time has a real problem and drops performance below the levels of the Proliant at 1000 concurrent connections.

Even 10.000 requests don’t seem to matter much to the Sun. The Lynx is doing good until 500 concurrent connections, rises again at 750 only to drop to the level of the Proliant at 1000 concurrent connections.

Now, we enter the realm of serving dynamic pages.

First thing to note is that when your intention is to get out a lot of pages fast, you don’t use PHP. Never. Just look at the numbers. Where we had a whooping 16.000 pages per second when serving static content, we now max out at around 270 pages a second. Ouch. As one can see, the Sun handles the job a lot better than the other machines.

At 2500 requests the Lynx tries to put on a fight but doesn’t quite make it.

Same situation as above.

Same situation again.

Still no change at 10.000 requests.

What I found strange in this run is the drop in performance shown by both the Sun and the Lynx. Also, the total number of pages served isn’t that much higher when using the keep-alive feature as was the case in the runs with the static page. Most likely that’s because caching doesn’t kick in for a dynamically created page that much and that the bottleneck is the PHP interpreter itself.

At 2500 requests the drop in performance is still visible. Otherwise there’s not much difference to the previous chart except that the Lynx is gaining again at 1000 concurrent connections.

The run with a total of 5000 requested pages shows the Sun to finally stop losing performance at higher rates of concurrent connections. From 25 concurrent connections onwards it’s a nice line at roughly 275 pages per second delivered. That’s a lot more than the Lynx does and a tremendous amount more than the Proliant is able to handle.

The only difference when requesting 7500 pages is that the Lynx shows an inexplicable drop at 750 concurrent connections. Since there was no other traffic on the network I’ll have to do some more investigating to learn what caused this. For the moment I’m willing to consider it to be a glitch in the tests.

Well, no real change at 10.000 pages requested. The Sun still outperforms the Lynx which in turn again drops in performance at the higher end of the concurrent connections tested.

The Files
If you’re interested in the complete set, you’ll find them here:
apachebench.xls.tar.gz (Excel format)
apachebench.ods.tar.gz (Open Document format)

Also, if you want to run the test on your hardware, here is the script I used to measure the performance of the systems:
apache_bench.tar.gz

To run it, you should create a directory below the one where you unpacked the archive. The name is not important but it’s good practice to have it reflect the client you are testing from. Next, edit the shell script runme so that the IP addresses used are the ones of your servers. Then change to the directory you created and start the benchmark using “../runme”. The script will create a lot of files in the current directory. To generate a tab separated file readable by Excel from that, you can use beautify.php which is also included in the archive. Have fun.

[ All posts about my experiences with the SunFire T2000 >>> ]

Incoming search terms:

• whocares sunny days
• load balacing apache test scenario
• readynas apache is running