The easy-math way to know if you can test your fundraising idea

Testing Tuesday: A weekly series on how to get the most of direct-response fundraising tests

Smarttesting

Everyone tells you that you should be testing your fundraising messages.

But should you?

Possibly not. Most fundraisers shouldn’t be testing.

Not because testing isn’t worthwhile, but because the large majority of organizations don’t have enough donors to give trustworthy test results.

For most nonprofits, the chance that test results will be untrustworthy are high. Almost certain. It would be like going to the doctor to test your blood for some condition, and finding out the chance of a false positive or false negative are more likely than an accurate result

You don’t want that for your health or for your fundraising!

Here’s a quick way to estimate if you have enough quantity to do a valid test:

  • Determine your testing quantity, that is, the total number of pieces you are going to mail or email.
  • Split that number in half, if it’s a two-panel test — thirds if it’s a three-panel, etc.
  • Project the likely response rate of the project. Make your best educated guess, erring on the side of pessimism.
  • Calculate what that response rate is in numbers for each panel.

If that final number is under 100, don’t test.

Because you don’t have the quantity to give you valid results.

Example:

Say you want to test whether a blue envelope or a green envelope will work best in a mailing to your donors. It’s your March appeal, generally an average performer.

  • Your mailing list is 5,000.
  • Each test panel will be 2,500.
  • You project 6% response.
  • That’s 150 responses per panel.

Go ahead and test! You have a chance of getting valid, statistically significant results that you can believe.

But let’s say you want to do the same test for a donor acquisition mailing to a list of 5,000.

Because it’s donor acquisition, you project 1% response. That’s 25 responses per panel.

Don’t test!

Projecting 100+ responses per panel doesn’t guarantee you’ll get statistically significant results. But if you’re below 100, you pretty much guarantee that you won’t get it.

The whole issue here is statistical significance, which is how you separate “signal” from “noise” in testing.

Read the full Testing Tuesday series


Comments

2 responses to “The easy-math way to know if you can test your fundraising idea”

  1. The ‘easy math’ here is handy and for sure you don’t want wasted time and money but I think this viewpoint is pretty much ONLY geared towards mail, and to be honest, possibly a damaging viewpoint as it relates to digital fundraising. Two reasons why this mindset doesn’t apply to digital and 1 more on fundraising testing in general:
    1) “No Cost” – it’s in quotes because there is time and opportunity cost but no hard cost and the time is quite small to invest in an email test. So what if you’re email test didn’t validate… what did you actually lose? Maybe it could validate… maybe you can see a directional lift… and the cultural win of testing and curiosity is far more beneficial in the long run than avoiding an ‘insignificant’ test.
    2) You can test ‘up funnel’ – imagine if you could know how many envelopes were opened? How many people read the letter? ALL of the letter? Looked at the reply device? Started filling out the reply device? These are all things you can measure and better understand in a test on digital that you can’t really in mail. So maybe you won’t get 100 donors but you can get 1,000 clicks which can be informative on what drives interest that could lead to donations.
    3) The goal is learning and growth not JUST significance. We teach/preach/practice test validity and use 95% BUT also believe that we aren’t producing academic journals but merely trying to better understand donors and deliver more results so anything over 80% IMHO is worth taking note of. There’s a 20% chance the result was false so keep that in mind but I’d rather test and know the 20% chance it was wrong and factor that into decisions than not test at all because I didn’t have enough volume to guarantee or have a shot at 95%.
    I’m guessing you’d agree with much of the above and a conversation around testing is much more complex and I respect your attempt to simplify it, truly, so keen to get yours and other folks’ thoughts here.

  2. The ‘easy math’ here is handy and for sure you don’t want wasted time and money but I think this viewpoint is pretty much ONLY geared towards mail, and to be honest, possibly a damaging viewpoint as it relates to digital fundraising. Two reasons why this mindset doesn’t apply to digital and 1 more on fundraising testing in general:
    1) “No Cost” – it’s in quotes because there is time and opportunity cost but no hard cost and the time is quite small to invest in an email test. So what if you’re email test didn’t validate… what did you actually lose? Maybe it could validate… maybe you can see a directional lift… and the cultural win of testing and curiosity is far more beneficial in the long run than avoiding an ‘insignificant’ test.
    2) You can test ‘up funnel’ – imagine if you could know how many envelopes were opened? How many people read the letter? ALL of the letter? Looked at the reply device? Started filling out the reply device? These are all things you can measure and better understand in a test on digital that you can’t really in mail. So maybe you won’t get 100 donors but you can get 1,000 clicks which can be informative on what drives interest that could lead to donations.
    3) The goal is learning and growth not JUST significance. We teach/preach/practice test validity and use 95% BUT also believe that we aren’t producing academic journals but merely trying to better understand donors and deliver more results so anything over 80% IMHO is worth taking note of. There’s a 20% chance the result was false so keep that in mind but I’d rather test and know the 20% chance it was wrong and factor that into decisions than not test at all because I didn’t have enough volume to guarantee or have a shot at 95%.
    I’m guessing you’d agree with much of the above and a conversation around testing is much more complex and I respect your attempt to simplify it, truly, so keen to get yours and other folks’ thoughts here.

Leave a Reply

What this blog is about

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Blog policies

Subscribe

Get new posts by email:

About the blogger

Jeff BrooksJeff Brooks has been serving the nonprofit community for more than 35 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com. More.


Archives

Blogroll

Categories


Search the blog

The future of fundraising is not about social media, online video, or SEM. It’s not about any technology, medium, or technique. It’s about donors. If you need to raise funds from donors, you need to study them, respect them, and build everything you do around them. And the future? It’s already here. More.

Recent Comments

About the blogger

Jeff Brooks has been serving the nonprofit community for more than 30 years and blogging about it since 2005. He considers fundraising the most noble of pursuits and hopes you’ll join him in that opinion. You can reach him at jeff [at] jeff-brooks [dot] com.