Trusting Test Results

Testing helps you work out the most effective approaches but only if you can trust the results. Testing was explained in Testing with A/B tests explained in the Testing to Find Effective Approaches blog posting. This post gives some pitfalls to avoid when creating tests.

There are two key principles to follow; have a sufficient number of responses and test only one thing at a time.

Have a Sufficient Number of Responses

If you look at any opinion poll and read the story carefully, you’ll see that they often give a margin of error. So the headline may be that political party A has 35% support whereas party B has 40%. But if you read the details it may be that there is a 5% margin of error, meaning that Party A actually has somewhere between 30% and 40% support, with Party B having 35%-45%. So suddenly the results don’t feel as trustworthy. Party B probably has more support, but you can’t be sure.

Anything with results or statistics will always have this problem. You can reduce the margin of error by increasing the amount of people involved – or the sample size, to use the jargon. So the results of a test with 1,000 responses will be more reliable than one with 500.

Test One Thing at a Time

Let’s say you have two potential Google Adwords adverts that you want to test; A and B.

 A says;

Quality Marketing Advice
Professional Marketing Advice for the Micro-business you can rely, on from a local supplier.

B says;

Low-cost Marketing Advice
Top-quality Marketing for the Micro-business at a price you can afford from a local company.

If A was the top-performing advert you wouldn’t know whether it was due to the ‘quality and professional’ message, or whether the headline grabbed people’s attention or whether it was the wording of the advert itself. There are too many variables.

A truly robust test would have four variations;

Quality Marketing Advice
Professional Marketing Advice for the Micro-business you can rely on, from a local supplier.

Quality Marketing Advice
Top-quality Marketing for the Micro-business at a price you can afford from a local company.

Low-cost Marketing Advice
Professional Marketing Advice for the Micro-business you can rely on, from a local supplier.

Low-cost Marketing Advice
Top-quality Marketing for the Micro-business at a price you can afford from a local company.

This approach allows you to test both headlines and both sets of wording simultaneously but you would also need to consider the number of responses.

Conclusion

It may not be feasible to have this degree of robustness in every test you do, but the closer to it you can get, the more you can rely on the results before committing.