What did you learn from that test?

Testing Tuesday: A weekly series on how to get the most of direct-response fundraising tests

Smarttesting

What did you learn from that test you just ran?

It’s not always obvious. Let me give you an example.

Let’s say you decide to test two different photos in your direct mail pack. You wonder whether a “sad” photo or a “happy” photo will do better. You have sufficient quantity. You’re disciplined, so everything else in the pack is identical, except for this one photo:

Photoa
Control: PHOTO A (Sad Old White Guy)

Photob
Test: PHOTO B (Happy Old White Guy)

So far so good. You wait a couple of months to get the results (or a couple of days if your test is online).

Let’s say that the test, PHOTO B, Happy Old White Guy, outperforms at a statistically significant level.

Hooray! You’ve learned something from your donors.

But what have you learned?

You have not learned that happy photos are better than sad photos.

You’ve only learned that this particular happy photo in this exact context performed better than that particular sad photo.

If you are going to be sending this exact direct mail piece repeatedly, you’ve learned something very valuable: With the new photo, this mailing will do better from now on.

But maybe you’ve uncovered something beyond this one mailing. In this hypothetical case, the less-expected photo did better. Usually, “sad” photos motivate more giving than “happy” ones. If your test shows otherwise, that should get your attention! Maybe there’s something you need to know.

In a case like this, you should test the same notion again, with other photos and in other contexts. It’s important to know whether that first test was a fluke (that happens!) or it was pointing to a fundamental truth.

Note that photos are especially slippery test subjects. You can never test the efficacy of photos in general — only whatever specific photo you have at hand. The specifics always trump the general.

With any test, you must know what you’ve really learned. And not draw over-broad conclusions That takes some careful thinking.

Read the full Testing Tuesday series


Comments

Leave a Reply

What this blog is about

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Blog policies

Subscribe

Get new posts by email:

About the blogger

Jeff BrooksJeff Brooks has been serving the nonprofit community for more than 35 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com. More.


Archives

Blogroll

Categories


Search the blog

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Recent Comments

About the blogger

Jeff Brooks has been serving the nonprofit community for more than 30 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com.

Blog Roll

someone’s blog