WinRed’s A/B Testing feature allows an organization to test two Upsell Flows by splitting traffic between them to see which one performs better.
This is a powerful tool designed to boost the effectiveness of your Upsells and a necessity for any fundraising program.
Follow the directions below to get started!
Create Upsell Flow and Test Variant
To run an A/B Test, you first need two Upsell Flows in your account that you want to test against each other.
Head here to create or modify an Upsell Flow. You can either create two different ones or duplicate the first flow and modify it by changing some of the details.
If you are new to creating Upsell Flows, you can learn more about creating one here.
When you have the two flows you want to test against each other, you are ready to create the test!
Set Up New A/B Test
Head to the Utilities tab, and then click on A/B Tests sub tab.
Click the Create A/B Test button and enter this information:
Test Name: Give your A/B test a name
A/B Test Notes: Include a brief description of the A/B Test.
This isn't required, but it's recommended, so your team has context to reference later.
Add a Control Upsell Flow and a Variant Upsell Flow to your test.
These are the ones from step 1. Search for them by their name.
When you have entered all of the required information, you are ready to launch your test!
Double-check that everything is correct. Note that you cannot change the Upsell Flows once you have started the test.
Click the Start Test button to begin your test.
Drive Donors to Your Test
With your test live, it's time to get some results.
Add the control Upsell Flow to donation pages. You can learn how to add Upsell Flows to donation pages here.
The test between the Control and Variant will fire anytime a donor gives on a page that has the Control Upsell flow attached.
Blast that page to your donors to start getting donations
View Results
With your test running, data will begin to populate, and if a winner is determined, you will see that appear in the results here.
A winner is declared when there is a statistically significant result. This means we are able to determine one upsell flow raises more per session. Tests run until a winner is declared, but if there is not a significant lift for either upsell flow, then the result is inconclusive, and the test stops after 90 days.
Here's a breakdown of all the results you will see:
Goal: All tests use dollars raised per session as the key metric.
Improvement: This shows how much better one of the two Upsell Flows is performing over the other. The larger the percentage difference, the more money that upsell is raising.
Probability to Be Best (PBB): This column shows you the probability of which test is the best of the two. This displays data after 101 sessions have occurred and is a sign of which test is more likely to be the winner.
Winner: When a winner is declared, it will be listed in your A/B Test tab.
An Upsell Flow wins because there is a statistically significant result indicating that donors give more per session on the winning Upsell Flow.
Tests need to have 1,000 sessions and run for a minimum of 24 hours before a winner can be declared.
Tests cease running after 90 days if there is not an obvious winner. You will see No Winner Declared if this happens.
A Test Won, Now What?
When your test is finished and you have a winner, here's what you can do next:
Apply the Winning Upsell Flow to All Pages
Use the Bulk Actions tool in your account to update all of your pages with the winning upsell flow
Run Another Test
Clone your winning Upsell Flow and make some changes so you can set up a new test. The key to optimization is to make sure you are always testing, learning, and improving.
Other Notes
Only donations that are made on a Double Donation, Donation, or Panel (Double Donation or an Upsell Card) Upsells count towards the calculations to determine Improvement, Probability to be Best, Dollars per Session, Avg. Contribution, Donations, and Raised.
Money Pledge and Recurring Donations made on Upsells ARE NOT considered or taken into account in the calculations or data that is shown to you in the Portal.
If an Upsell only has one of these two Upsells or only these two Upsells, only the Sessions data will update with each visit to the Upsell Flow
Agreement by the donor to sign up for non-monetary Upsells (Volunteer, Mobile Opt-in, Create an Account, or Custom Content) do not count as a donation.
If an Upsell Flow only has one of these four Upsells or only these four Upsells, only the Sessions data will update with each visit to the Upsell Flow
Refunds and Disputes ARE NOT considered or taken into account in the calculations or data that is shown to you in the Portal
If a donation is made on a Double Donation, Donation, or Panel Upsell (Double Donation or an Upsell Card) Upsell, then it counts towards calculations and data you see
If that donation is later refunded or disputed, the calculations and data do not refresh to reflect the refund or dispute.
Business/PAC donors do not see Upsells, so those donors will not be part of the A/B Test stats.
Questions?
As always, if you have any questions, feel free to reach out to our team by opening a WinRed chat box at the bottom right of your screen!