Ode to Taguchi

Good marketing professionals are capable of creating compelling messages. They know where to place that message so it will reach the right audience at key moments. Decent and capable marketers know how to measure results effectively to prove success and, in some cases, failure.

Even better marketers understand that not every idea will be a winner. They know that sometimes plans don’t pan out. Really talented marketers know that they don’t have all the answers, but they do know how to find them.

“He who knows best knows how little he knows.”

– Thomas Jefferson

In fact, when speculating on what creative will resonate, what call to action will generate the most activity and what elements will encourage the customer to take the next step, the only voice that really matters is that of the customer. The best marketers create testing programs that help them to listen to the customer, who will always point them in the right direction. Simply put, the smartest marketing minds in the world test everything.

From simple to extremely complicated, there are many kinds of tests you can run. All these can be divided into two primary categories:

A/B Split Tests
An A/B Split helps you determine how one specific element will impact a customer’s decision (e.g. Subject Line A vs. Subject Line B, red button vs. blue button, % off vs. $ off).

Multivariate Tests
More involved than simple split tests, multivariate tests take into account that it’s not always just one element that makes the difference. Rather, the proper combination of elements drives the best results.

The one drawback to multivariate tests is the time required to get a statistically significant result. To use a pay per click text ad as an example, let’s say you want to test two values for three separate variables: Headline, Description and Display URL. To run a full multivariate test, you would have 8 different ad combinations.

  2 Headlines
x 2 Descriptions
x 2 Display URLs
8 Ad Combinations

For campaigns with limited traffic and/or budget, getting some real results from this testing effort could take weeks if not months. That’s where Genichi Taguchi comes in (not to be confused with a baseball player with the same name).

Mr. Taguchi is a Japanese-born engineer and statistician. During his career he has put his incredible brain to work for the Imperial Japanese Navy, the Japanese Ministry of Public Health and Welfare, Princeton University, Bell Labs, and the Japanese Standards Institute just to name a few. Over the years, he developed a statistical experiment model now known as the Taguchi Method. Originally developed for the purpose of improving the quality of manufactured goods, the Taguchi Method allows for a more efficient (albeit controversial) means of finding out what combination of elements will produce the best result. By testing specific combinations against one another, marketers can compare the results of each combination and determine what is the best combination based on that comparison. Let’s use an example…

Here’s what the PPC ad test would look like with a full factorial multivariate test.

Using the Taguchi Method, the test would look like this.

Once you have reliable response rates or other results from your test, you can apply a formula to determine which combination of elements is the best. Since marketers aren’t always the best mathematicians, just use this tool to figure it out. To demonstrate further how this works, let’s apply some dummy data. The percentages below represent click through rate for our example.

By applying the Taguchi Method formula, the winner is : AAB. Yes, a combination that wasn’t even tested is determined to be the best. Why wasn’t the ad with the best click through rate simply declared the best combination?

• Well, BAB did have a click through rate (CTR) of 9.0%.
• You can see that it shares the same Description as AAA which had the second highest CTR at 8%.
• However, it also shares a Headline with the worst performing ad combination, BBA.
• Thus, AAB provides the best headline, description and display URL combination. In this case, the description has the highest influence on the success of the ad.

Admittedly, the Taguchi method is not perfect and has its share of well-respected critics. It relies on a number of assumptions, and there are always other factors at play when running a test. Also, by not testing every single combination, you cannot be absolutely sure that you are selecting the actual winner. However, while not the “best” method, it is an incredibly useful process for improving results.

Here are some metrics and elements that you can use with your multivariate tests. As you can see there are several different ways to apply the Taguchi Method to your digital marketing campaigns:

Paid Search Advertisements (click through rate)
– Headline
– Description
– Display URL

Online Display Advertisements (click through rate)
– Creative (Message, Colors, Graphics)
– Call to Action
– Ad Size/Format

Landing Pages (conversion rate)
– Text Elements (Headline, Descriptive Copy, Support Copy, Bullets),
– Graphic Elements (Pictures, Diagrams, Icons)
– Promotion (Type, Offer/Amount/Level)
– Call to Action (Text, Buttons, Placement)
– Page Layout (Design Affects, Form Placement, “Above the Fold” Items)

Contact / Info Request Forms (conversion rate)
– Headine & Lead-In Text
– Fields (Number, Alignment, Requirements)
– Submit Button (Text, Color, Design)

Email Marketing (open rate)
– From Name
– Subject Line

Email Marketing (click through rate)
– Headline
– Copy
– Call to Action
– Graphics
– Layouts & Colors
– Promotion
– Delivery Time/Day

What other elements have you tested in the past or are you planning to test in the future?