Experiment design

Marketers need to be exhaustive with their creative testing, or they’ll risk missing a winning variation. But even with just a handful of variables to test, you can quickly reach an impractical number of combinations. Launch too many experiments at once and it’ll take too long to reach statistical significance. For example say you wanted to test 4 elements of an ad – headline, description, image, call-to-action – and you had 3 options for each. To test every possible combination would give you 3 ^ 4 = 81 variations to test. Assuming an average $10 CPM, 1% CTR, and detectable effect of +20%, you would need to spend $1,000 a day for 32 days in order to reach statistical significance, as found using a test duration calculator. This may potentially be too long to wait, or too much to spend, meaning you’d have to scale back the ambition of your testing plan.

Thankfully there’s a technique, popularized by Taguchi, called “Orthogonal Array Testing”. This method moves away from the traditional approach of testing one factor at a time and, instead, tests many factors at the same time but in a more strategic way. Taguchi’s approach dramatically reduces the number of experiments required, bringing down costs and getting results quicker. Bob Moesta, who worked under Taguchi, said of the technique “The reality is because I’m changing everything simultaneously, I learn so it becomes more reproducible and I can manage the tradeoffs”. The method works by testing independent pairs of parameters once each to see how they interact. Every test gives you maximum information gain, allowing you to infer what would happen if you test the rest of the combinations.

In practice it works as follows:

#1) Decide the number of variables that will be tested

#2) Decide the maximum number of values that each variable will have

#3) Find a suitable orthogonal array with the smallest number of runs

#4) Map variables → factors, and values → levels in the array

#5) Run the stated experiments and interpret the results

Let’s take our example from earlier. We were testing 4 elements, or factors, and we had 3 options for each, which are the levels. We calculate an orthogonal array or check a lookup table for an orthogonal array that’s labelled 3 ^ 4, and get the following:

Remember if we were to test all combinations we’d have to run 81 tests. However this orthogonal array is pointing us to the 9 tests that contain the most information, based on pairwise interactions. Note if there isn’t an exactly matching orthogonal array, take one with slightly more columns and delete columns from the right until you have the right amount. We can now map the variables to factors, and values to levels to give us our testing plan:

Once this experiment has run, which should only take 4 days rather than over a month, we can infer what combinations of factors are important. First we look at “Single Mode” anomalies, or unexpected high or low performance due only to one parameter. For example if variations 7, 8, and 9 failed, we can expect that Headline option C is causing poor performance. Next we look for “Double Mode” anomalies, where two specific parameters interact together. Because each pairing only occurs once, we’re looking for any one variation that outperforms the rest. If more than two parameters are interacting, we can’t tell with this array, because only pairs are represented. However when we find a winner, we should at least be in the right region for further testing. This allows us to cover a broad surface area in a short space of time, and prioritize the combinations of variables and options with the most promise.

About the Book 📕

← Back to Essays 📝