Load Testing With Apache Benchmark and Siege

Introduction

What is Testing?

Apache Benchmark (ab) and Siege are two tools that help us to accomplish load tests. Load tests are important elements in conducting stress and performance tests. Roughly speaking...

...load testing is when we steadily increase the load (usually requests) upon a system until it reaches a given target load. Often this is the max anticipated load (number requests to various resources at max traffic, for example), but can also be the sustained average usage of the system over time, or even zero load.

...stress testing is when we try to find the point at which the system breaks by overloading it with requests, removing certain components, and otherwise fucking with it. This helps us see not only how a system breaks (and thus find how to improve it), but also how gracefully it recovers after failure.

...performance testing is the process of establishing various benchmarks and seeing if our application can meet them. Often times performance testing is carried out through a series of load tests.

For more info on the differences between these types of tests, check out http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html.

Setting Benchmarks

We are primarily concerned with performance testing. At the beginning of a project, certain benchmarks should be set from conversing with a client.

What does peak traffic look like on their existing site?
What extra requests could we expect from new features? (ex: a feature that communicates with an external resource will consume resources differently than a new static page)
How fast do certain pages need to load?
How responsive do certain features need to be? (ex: do notifications need to be sent from a site immediately after an action, or can we batch them for later delivery?)

Once we've made some estimates on the performance benchmarks that we need to hit, we can design a series of load tests to see how our application behaves as it strives to meet those benchmarks.

Warnings

DO NOT perform tests on Pantheon Dev/Test servers; although Test is a rough approximation of Live (albeit with fewer resources), if you overload Test, Pantheon may shut down your Test and Dev instances.
DO NOT perform tests on a live site during peak hours.

Installing Tools

Apache Benchmark

"ab" should come pre-installed on almost any server or computer.

Siege

On your Mac, definitely use Homebrew to install Siege (brew install siege). If you haven't installed Brew, check out http://brew.sh.

On other *nix environments, use your normal package manager to handle the installation. On Kalabox, it's a simple "apt-get install siege".

Basic Anonymous Load Testing

Apache Benchmark

# send 10 requests by 5 concurrent users to the url
ab -n 10 -c 5 http://www.kalamuna.com/

# 500 requests by 50 concurrent users
ab -n 500 -c 50 http://www.kalamuna.com/

# authenticate when blocked by .htaccess or other auth
ab -n 10 -c 5 -A kala:kala http://www.kalamuna.com/

# explicitly accept encoding for better results
ab -n 100 -c 50 -k -H 'Accept-Encoding: gzip,deflate' http://www.kalamuna.com/

Documentation

http://httpd.apache.org/docs/2.2/programs/ab.html

Siege

siege --concurrent=50 --reps=100 http://www.kalamuna.com

Basic Authenticated Testing

Apache Benchmark

To have ab make its requests as an authenticated Drupal user, you'll need to use the "C" (cookie) flag with a valid session cookie. To do this...

Log in as a user with the proper access for the page you want to test with ab.
Open up the Chrome dev tools and look at the "Resources" tab. Copy both the key and value for the session.
Insert those values in the format below as arguments for the "C" flag (see below)

To make sure that you're receiving the correct content, you can set the verbosity level to "4" to see some of the information returned by ab's requests:

You can add helpful snippets of code to your template files or elsewhere to make sure requests are being sent as an authenticated user, more on that here.

Siege

Simulating Real User Load

The above stress testing methods are great for simulating a bunch of people hitting a single page, but what about stress testing the application as a whole? Some errors don't appear unless multiple users are requesting different resources all at once. Case in point: a mysterious seg-fault error was happening on one of our sites, accompanied by a cryptic PHP Fatal error in template.inc. Users were intermittently experiencing 500 responses, but it was difficult to ascertain the pattern in the error's appearance. Turns out that a number of curl_exec calls, used to interact with an external API, were taking longer than expected to return results. For each curl_exec call, a new php-fpm worker was spawned. Since most NGINX configurations only allow a finite number of these workers, at a certain point we were hitting the limit, causing other areas of the application to fail when they needed a spare worker to process a page.

The solution: use Siege to hit multiple pages concurrently. Siege allows you to input a file with a number of URLs in it as an argument instead of a single URL. Something like the following will allow you to better simulate real user load on a site:

The some-urls.txt file is just a simple list of URLs on newlines:

That should allow you to blast those two URLs concurrently. By default Siege will hit them in the order you have provided them in some-urls.txt; insert the "–internet" flag to have Siege make requests to the given URLs randomly.

If you're looking to better recreate the usage pattern of real users, there are some folks who have tried creating scripts/other tools for generating URL lists. I haven't tried anything with these, but you can checkout https://drupal.org/node/48894 and report back!

Apache Benchmark Example Results

TODO: Describe how to read/use this information. Add some Siege examples.

Resources

Performance Testing on Pantheon