DRAFT

Providers

Optimize.ly

Optimize.ly vs Visual Website Optimizer

There are a few different A/B testing tools out there. Two of the top ones are Optmize.ly and Visual Website Optimizer(VWO). I believe that VWO integrates deeply with google analytics, however I do not have experience with it. Optmize.ly also has GA integration, but it takes a bit of setup and I'm not sure how well it integrates with goal funnels. However, Optimize.ly has a very quick and easy setup, with lots of options for A/B tests, so that's what I use.

Integrating Optimize.ly with drupal

There is a contrib module for Optimize.ly which is simple and works well. Essentially it just embeds the tracking js into your drupal javascripts array which gets output onto the page. There are two main configuration options: (1) Add your optimize.ly account number, and (2) Configure which paths the code should show up on. I usually just have the code show up on every page. At the very least, you'll need to the optimize.ly tracking js to be present on the page you are changing the design on, and the goal page(s).

Creating the Experiment

The first step with doing an A/B test is to create an experiment. As experiment is usually defined as one "set" of changes you want to test. For example - if you want to test the effect of your new homepage design on bounce rate, and the effect of your new shopping cart design on conversion rate, you'd create two experiments. If you wanted to test the effectiveness a navbar change globally across mutliple pages on user engagement, that would be one experiment, even though it spans multiple pages. Note: You may need a higher level subscription to perform multi-page experiments.

Creating the Design Variations

In Optimize.ly, the two primary ways to A/B test are to (a) create two pages on your site, then tell the tool to use these for the test, and (b) use the WYSIWYG editor to make the tweaks the to the page you'd like to use for the experiment. You would use (a) for situations in which the markup is changing dramatically, and (b) for simpler experiments (i.e. changes in navigation, imagery, etc). But don't underestimate option (b)... it's fairly impressive and allows you to add you're own markup, javascript, and css.

Defining Goals

In any case, you'll also need to set up goals for your experiment. Goals are what define which of the designs in the test "wins." A goal is usually that a user reaches a destination page (eg. cart/checkout/success) during the course of their session, but can also be triggered by canned events you can set up using the editor interface ("user clicks X button") or custom javascript events which trigger an API callback.

Running Experiments

Once you've set up your experiment, you'll need to Start the experiment in order for it to begin gathering data. It's not uncommon that once you begin the experiment to will find that some piece of the experiment isn't working as expected due to a configuration/setup problem, so I recommend setting up and letting it run for a while while continuing tweak it. Once things are working as expected, flush the results to get the old, bad, data out and just let it run. Optimize.ly will handle the statistical analysis for you, and let you know when there is a clear winner.

Google experiments

http://www.netstudio.gr/en/blog/how-use-new-google-content-experiments-ab-testing-platform-optimize-your-drupal-site

Crazy Egg

https://drupal.org/project/crazyegg

Implementation details

Drupal can perform A/B against dedicated URLs, but it can be more complex to test variations in layout or the display of page elements that use Panelizer.

Panels pages support variants.

panels support

Kalawiki

AB testing