How to learn from direct-response testing

It’s so cool that when we use measurable direct-response media like direct mail and email we can test. We don’t have to take anybody’s word for it about what works and doesn’t.

But not all tests teach. In fact, a lot of tests are utter wastes of time and money. Some even lead fundraisers badly astray because they’re poorly set up and not well analyzed.

The MarketingExperiments Blog has some basic help for setting up good tests, at A/B Testing: Example of a good hypothesis.

The main point is that, as with any scientific experimentation (which is exactly what testing is) you must have a clearly stated and well-understood hypothesis:

… your hypothesis should follow a structure of: “If I change this, it will have this effect,” but should always be informed by an analysis of the problems and rooted in the solution you deemed appropriate.

It’s entirely possible your hypothesis is wrong. That’s okay. If have a clear hypothesis, you’ll get valuable information from the test, either way.

Let’s say you have a reply device that’s largely blue. You wonder if it might do better if it were red. Your hypothesis: Red design will pull better results (response, average gift, or both) than blue.

With a clear hypothesis, you can read the results without confusion, no matter how the test turns out.

Often, though, with no clear, written, agreed-on hypothesis, you have no clear read on the results:

1. Person X thought you were testing brand color vs. non-brand color.
   
2. Person Y thought you were testing warm colors vs. cool colors.
   
3. Person Z thought you were testing American style vs. Canadian style.
   

With all those different hypotheses, you could have a mess of interpretations of the results. And end up knowing less than you did before you started.

Testing is science. Treat is that way, and you’ll learn.

Tomorrow: testing that makes you dumb.