Direct response testing that makes you dumb

Yesterday, we talked about the importance of a clear hypothesis when you do direct-response testing. Today I’ll show you how a lack of clarity can trip you up.

Early in my career, I had a client that, just before I arrived, had tested “organizational copy” against “story-based” copy. Organizational had outperformed, so we had clear marching orders to write “organizational copy.”

You’re probably as puzzled by that as I was. It seemed to say donors were more responsive to the organization talking about itself than they were to a story that illustrated the need.

So I wrote fundraising like that.

And it didn’t work very well.

Finally, I did what I should have done in the first place: Went back to the test and actually read the two letters. Was I ever surprised.

The “organizational” version was all about the call to action — not particularly the organization. The “story” version was dominated by a story — a decent story, but it much less clear about the call to action.

Having a clear offer beats just about anything else you might do, including a story. But the label of “organizational” obscured that reality and sent us down an expensive and mistaken path. I’m sure it made complete sense to whoever came up with it, but since it was never written in the form of a clear hypothesis, there’s no way to know what they were actually aiming at testing.

When you’re testing, always put the hypothesis in writing. And always make sure everyone involved has the same understanding of what they’re intending to learn.

Otherwise, you’re better off not testing at all.