Three Ways A-B Testing Will Improve Your Marketing. (Part 2) Avoid Misdirected Testing.

Last week we decided that any testing framework needed to have a specific destination in mind – your objective.  While it is reasonably straightforward to match your testing framework to your overall Marketing objectives, it is easy to get misdirected and lose track of your destination.

If your car has a GPS navigation system and if you’re anything like me (I always have a better route), you have heard that voice in the GPS say, “Turn around at the nearest U-turn!” Repeatedly. Mindlessly. Until you just turn it off. (Or, in my case, the GPS finally gives up, decides I’m right, and recalibrates the new course.) You’re A-B testing framework should be like that voice in the GPS, incessantly reminding you that you have left the prescribed course. How? Here are some guidelines to help you understand when your testing is off-course.

Guideline #1: A-B test results are not extensible and repeatable
Subject Line testing often falls into this category. If your testing is not extensible and repeatable, the test is only applicable to that particular email. To repeat a lesson we learned in {Demand Gen Brief} last week, categories of tests, rather than specific tests should be used like this:

We would apply scientific method to create a series of hypotheses to test these assumptions to ultimately create a rule by which all future subject lines are created, such as:

1.     Subject lines should be less than 35 characters.

2.     Subject Lines should include our company name.

3.     Subject Lines should contain the recipient’s first name.

Guideline #2: A-B tests are not actionable

What’s the point of performing a test if you can’t action the results? An example of such a test would be to perform an A-B test on CTA button colors only to find that purple wins by a landslide. Problem? Purple is your main competitor’s color and your brand standards prohibit its use in any way. There are, however, some very interesting takes on this guideline. For example, “best practices” are to not use some key words in your subject line, because they can trigger email client SPAM filters. Or not. What if 50% of your emails got sent to SPAM filters because you used the word free in the subject line, but the remaining 50% showed a 1200% open rate increase over the next best subject line? Would you break “best practices” and go with free? Of course you would, unless…

Guideline #3: A-B tests are specific to only one part of the equation

Remember, each test in your framework is designed to test a specific metric, and each metric measures only a part of the journey from Prospect to Closed/Won. In the previous example, using the word free in the subject line increased your open rates by 1200%. Great, but what happened next? Did those opens turn into clicks? Did those clicks turn into MQLs, SQLs and, ultimately closed business? Your testing framework should be looking at the entire lead lifecycle to determine how each action contributes to the progress of the entire demand funnel. Let’s look at another example.

Open                    Clicks                       Convert to MQL                   Convert to SQL                   Closed/Won

1200x.5=600      .01x600=6               .50x6=3                                  .5x3=1.5                               .5x1.5=.075

100                       100x.25=25            .50x25=12.5                          .5x12.5=6.25                        6.25x.5=3.125

In the end, that great open rate using the word free only resulted in opens, not clicks. Using the exact same conversion rates from MQL through close, we find a 400% improvement in closed business not using the word free in the subject line.

Change the conversation.

Again, A-B testing must be performed within a holistic framework that considers the entire demand funnel and what each tested step contributes to the whole. Unless you get paid for people opening your emails, the 1200% increase in open rates does not serve your organization’s overall objectives.

Notes:

Testing must be performed within an overall framework with a specific objective in mind.

Testing must consider both the specific action being tested and its overall contribution towards the overall objective.

Again, “best practices” for company A are not necessarily best practices for company B.

Next week, we’ll look at how to tie your testing framework to demand funnel progression, and why it is critical to build your framework that way: Three Ways A-B Testing Will Improve Your Marketing. (Part 2) Into the Vortex.