How to learn from direct-response testing

It’s so cool that when we use measurable direct-response media like direct mail and email we can test. We don’t have to take anybody’s word for it about what works and doesn’t.

But not all tests teach. In fact, a lot of tests are utter wastes of time and money. Some even lead fundraisers badly astray because they’re poorly set up and not well analyzed.

The MarketingExperiments Blog has some basic help for setting up good tests, at A/B Testing: Example of a good hypothesis.

The main point is that, as with any scientific experimentation (which is exactly what testing is) you must have a clearly stated and well-understood hypothesis:

… your hypothesis should follow a structure of: “If I change this, it will have this effect,” but should always be informed by an analysis of the problems and rooted in the solution you deemed appropriate.

It’s entirely possible your hypothesis is wrong. That’s okay. If have a clear hypothesis, you’ll get valuable information from the test, either way.

Let’s say you have a reply device that’s largely blue. You wonder if it might do better if it were red. Your hypothesis: Red design will pull better results (response, average gift, or both) than blue.

With a clear hypothesis, you can read the results without confusion, no matter how the test turns out.

Often, though, with no clear, written, agreed-on hypothesis, you have no clear read on the results:


  1. Person X thought you were testing brand color vs. non-brand color.
  2. Person Y thought you were testing warm colors vs. cool colors.
  3. Person Z thought you were testing American style vs. Canadian style.

With all those different hypotheses, you could have a mess of interpretations of the results. And end up knowing less than you did before you started.

Testing is science. Treat is that way, and you’ll learn.

Tomorrow: testing that makes you dumb.


Comments

Leave a Reply

What this blog is about

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Blog policies

Subscribe

Get new posts by email:

About the blogger

Jeff BrooksJeff Brooks has been serving the nonprofit community for more than 35 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com. More.


Archives

Blogroll

Categories


Search the blog

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Recent Comments

About the blogger

Jeff Brooks has been serving the nonprofit community for more than 30 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com.

Blog Roll

someone’s blog