How to make sure you learn what you really learn from your test

Testing Tuesday: A weekly series on how to get the most of direct-response fundraising tests

Smarttesting

We were looking at some very odd direct mail test results. They seemed to show a result exactly at odds with what experienced people know to be true.

The test showed that an “organizational” approach in the message convincingly out-performed a “personal” approach.

You’re probably thinking exactly what I was: What the heck?

I assumed (as did everyone else involved) that “organizational” meant it emphasized the organization’s role in the topic at hand, and “personal” meant it emphasized the donor’s role.

So for the past two years, the org had been doing “organizational” fundraising. They carefully described how their programs worked. They featured staff members as the amazing people they were. The message was “Donate now so we can do the things we do so well.”

It wasn’t going well.

The retention rate was low and dropping. New donor campaigns were not sustainable.

They were in a revenue death-spiral.

That’s when I asked to look at that old “organization vs. personal” test. Not just the results, but the actual packs.

Here’s what I discovered: The two versions were quite similar, best-practice direct mail packs.

  • The “organizational” version had a very specific, simple clear offer. It asked the donor to give to make something specific and important happen.
  • The “personal” version had a chatty friendly tone, a lot like a letter to a friend — but no specific offer.

The labels “organizational” and “personal” weren’t exactly wrong, but they sure were ambiguous. They had no doubt come from a less-experienced member of the team, and were never checked clarity, and then became received truth.

Those labels were misinterpreted for two years and led the organization to do ineffective fundraising for two years. It cost hundreds of thousands in lost revenue.

We quickly changed the approach. Fundraising results turned around immediately. But the lost donors meant less revenue for the next several years.

Lesson: When you test, have a clear hypothesis. Make sure there is shared clarity of what you’re testing and what you call the things you are testing.

Testing is science. You develop a hypothesis (like “a chatty tone might improve results, even without an offer”) experiment, and learn. Part of that hypothesis is making sure you label your test in a way that you’ll learn what you really learned.

What they learned in this case — but didn’t know it — was that an offer was more important than a friendly writing tone.

We soon after tested a campaign that had a friendly tone and an offer — and I bet you can guess what we learned:

Yep. A friendly tone with an offer worked better than either one of those things alone.

Testing is about learning and improving. It takes a methodical, disciplined approach. And clarity about what you’re testing. Without that, you are far better off not testing.

Read the full Testing Tuesday series


Comments

Leave a Reply

What this blog is about

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Blog policies

Subscribe

Get new posts by email:

About the blogger

Jeff BrooksJeff Brooks has been serving the nonprofit community for more than 35 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com. More.


Archives

Blogroll

Categories


Search the blog

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Recent Comments

About the blogger

Jeff Brooks has been serving the nonprofit community for more than 30 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com.