It takes more than a bright idea, a fun plan, or hopeful intentions to run a successful promotional campaign. It takes an A/B test.
Marketers must be both strategic and creative when running a promotional campaign. Often, people will depend upon their ‘marketing intuition’ to make decisions, and that can prove to be problematic thanks to overconfidence and availability bias.
A/B testing is used in multiple areas within a marketing mix, but it is often used for:
- Identifying the audience marketers want to target
- Creating a campaign that meets the demands of said audience
- Measuring the campaign’s effectiveness
According to Hubspot, the benefits of A/B testing can include:
- Increased website traffic
- Higher conversion rates
- Lower bounce rates
- Lower cart abandonment
12 A/B Testing Best Practices Digital Marketers Must Know
A/B testing might come off as a simple task, and it can be, but there are a few general guidelines you may want to follow.
Below you’ll find 12 A/B testing best practices.
1. Understand the Basics of A/B Testing
The goal of A/B testing is to determine what digital marketing efforts work best for a given audience. It is called A/B testing because it refers to condition A and condition B: Marketers will tweak a piece of content or a message to determine which factors work best.
Condition A is generally the control, and condition B is generally the variation.
A/B testing happens in two different stages: the exploration stage and exploitation stage.
During the exploration stage, a single audience is randomly chosen and tested. Then, 10 percent of that audience is used to run the A/B test. One half of the 10 percent will receive version A, and the other half will receive version B.
In the exploitation stage, whichever version is most successful will be given to the remaining 90 percent of the entire audience.
There are a variety of tactics and processes you can use to run A/B tests, but Optimizely provides a basic outline:
- Collect data
- Identify goals
- Generate hypothesis
- Create variations
- Run experiment
- Analyze results
Stuart Frisby provides excellent insights on A/B testing:
2. Design a Strong Hypothesis
A/B testing is a type of controlled experiment, which means you need a well-designed hypothesis. While you should definitely determine whether condition A or condition B is more successful, you should also determine why.
For example: “Because we observed a plateau in conversions and received negative feedback from customers on website usability, we believe that placing the call-to-action at the top of the webpage will increase the amount of backpacks sold. We will know this is true when we see higher profits and obtain more conversions.”
3. Avoid Running Concurrent Tests
If you run more than one test at the same time with the same customers, you can contaminate the results of your A/B tests. Meaning, one A/B test will affect the perceptions of another A/B test and neither of their results will be accurate.
You should only test one factor at a time during an A/B test.
Let’s say you’re running one A/B test on two different colored buttons: one is blue and one is green. At the same time, you are running an A/B test on where you place your call to action: one at the top and one at the bottom of the page copy.
Running both of these tests at the same time could result in some users seeing the blue button with a call to action at the top of the page, the blue button with a call to action at the bottom of the page, the green button with a call to action at the top of the page, or the green button with a call to action at the bottom of the page.
This would make determining the success of either of the tests difficult because you wouldn’t be able to determine which variable was influencing people to perform the desired action.
4. Test the Right Factors
What is your bottom line?
[Tweet “In terms of A/B testing, your bottom line should focus on profit, not design. “]Do you want more people to buy your product? Do you want more people to sign up for your email list? Do you want people to stay on your website longer?
Determine what it is you want to accomplish through A/B testing.
5. Select the Correct Sample Size
You can’t get reliable results if you don’t test an appropriate number of people. It isn’t easy to determine the perfect sample size for your test, but the good news is there are A/B testing tools for that.
Optimizely’s Sample Size Calculator can determine your ideal sample size if you fill in your baseline conversion rate, minimum detectable effect, and statistical significance.
By selecting the right sample size, you will get more accurate results.
6. Run A/B Testing for a Full-time Period
A/B test results can vary depending upon days of the week.
Let’s say you’re running the exploration stage of an A/B test on Monday through Wednesday then running the exploitation stage Friday through Sunday. The results from the exploration and exploitation stages could be drastically different because of the weekend.
In general, it is best to run A/B testing for a full week, starting and ending on the same day of the week, at the same time of day. This way, you can get a good mix of results that are more likely to be accurate.
It can also be helpful to run the test at multiple times of day depending on what kind of promotional campaign you’re running. For instance, email marketing campaigns are generally released to all recipients at a specific time of day, but it can be beneficial to try different times to see which performs best.
Neil Patel shares a helpful video:
7. Control Seasonality
You should run exploration and exploitation stages at the same time of year, so you can rule out seasonality.
For example, if you run an exploration stage right before the holiday season, you should only run the exploitation stage right before the holiday season. Time of year can change everything.
8. Don’t Ignore Small Successes
What might appear as a small increase in return during an A/B test could result in tens of thousands of dollars in profit.
Pay close attention to your sample size in comparison to changes in percentage.
9. Recognize Your Local Maxima
You can only make so many tweaks and changes to your website before you’ve optimized your content as much as you possibly can. At this point, you’ve hit your local maxima. The local maxima is essentially the optimal form of your content as it currently is. When you hit the local maxima, you’ll find that changes to your website aren’t making much of a difference.
At this point, you may want to consider using a different design altogether. Don’t be afraid to shoot for a better design.
A local maxima looks something like this:
10. Use an A/B Test Calculator
By using an A/B test calculator, you can check the statistical significance of your results.
Statistical significance proves that your data is reliable. In other words, you can throw around numbers all day, but if they aren’t statistically significant, they mean nothing.
According to Neil Patel, “For meaningful results from meaningful data relationships, don’t stop running your test until you reach a statistical significance of 95%-99%, which simply means that you are 95%-99% confident that your outcome is valid.”
You can use an A/B test calculator to determine the reliability of your data:
- The Calculator – testing sample size and response
- Visual Website Optimizer – testing visitors and conversions
- Splitly – testing sessions and conversions
- AB Testguide – testing visitors and conversions
- Neil Patel – testing visitors and conversions
- A/B Test Calculator – testing participants and conversions
Each of these A/B test calculators will tell you whether or not your data is statistically significant, helping you understand the potential performance of your findings.
11. Use A/B Testing Tools for Extra Help
There are many A/B testing tools available online. Below you’ll find a list of some of the most popular:
- Optimizely
- VWO
- Convert Experiences
- SiteSpect
- AB Tasty
- Evolv
- Google Experiments
- Qubit
- Adobe Target
- Unbounce
- Optinmonster
- Qualtrics
- Mailchimp
- HubSpot
- Salesforce
- Splitly
With the help of A/B testing tools, the A/B testing process is much simpler. Many functions can be performed automatically at the click of a button.
These tools can be an excellent starting place for A/B testing. These tools are designed to work for every company, not yours specifically. For this reason, some professionals suggest that it is wise to develop your own A/B testing tools once you have the capability to do so.
12. Try Crowdsourcing
It is ideal to use an audience you’ve already gathered in your A/B test, but if you don’t have an audience large enough to run a proper test, you can use crowdsourcing communities to purchase an audience.
Crowdsourcing communities include:
A/B Testing Examples
To get started on your A/B testing journey, here are a few A/B testing examples.
1. Humana, a healthcare company, ran an A/B test for two banners on their homepage. Condition A contained lots of text, while condition B had less text and clearer calls to action.
Condition B hit a 433 percent increase in click throughs.
2. HubSpot, a digital marketing software, completed an A/B test to determine whether a sign-up form embedded inside of a blog post or a link to a separate sign-up page resulted in more email sign-ups.
Hubspot found that the embedded form performed 71 percent better.
3. Electronic Arts, a media company, wanted to increase sales for one of their games: SimCity 5. So, they ran an A/B test. Condition A showcased a purchasing screen that featured their two product packages along with a banner offering $20 off on the customer’s next purchase if they pre-ordered. Condition B showcased the same two packages but no pre-order discount.
Condition B was more than 40 percent more successful than condition A.
4. ComScore, a media evaluation software provider, tested their logos and testimonials to increase leads on their product landing page and ended up increasing generated leads by 69 percent.
5. Secret Escapes, a hotel booking service, doubled their conversion rates by A/B testing their mobile sign-up pages.
A/B Testing in the Classroom
Applying these concepts in the classroom can be difficult, considering A/B testing requires the use of an actual company’s website, email, or ad campaign. But, with the help of the following classroom application ideas, your students can get a feel for A/B testing without access to a real website.
- Have pairs of students run an A/B test of their choice, beginning with the formulation of a hypothesis. Then have them create a mock-up of condition A and B on a sheet of paper or whiteboard. For each pair of students, provide some form of voting to determine which condition would most likely perform best. (This does not mirror actual A/B testing, but can help students gain a basic understanding of it.)
- Have students create surveys with free tools like SurveyMonkey or Qualtrics in which they have only one question testing two variations of a call to action. Have the students share their surveys with classmates and friends. Then determine which call to action was more successful.
- Use Stukent Business Analytics Simternship™ to simulate a realistic A/B testing experience for students.
Stukent’s Business Analytics Simternship can provide your students with real, hands-on experience in all things marketing analytics. There are specific scenarios that require students to solve real problems with real data, complete A/B testing, perform machine learning techniques, and more.
Request free instructor access to the Stukent Business Analytics Simternship™.
For additional instructor resources and lesson plans, request free instructor access to the Stukent Marketing Analytics Simternship™. It covers all the primary marketing analytics concepts in one digital textbook.
Try New Things
A/B testing is meant to continually improve digital marketing efforts, so don’t be afraid to try new things. After all, you don’t know what you don’t know.
Marketers must be both strategic and creative when running a promotional campaign. It takes more than a bright idea to run a successful promotional campaign. It takes proper testing and experimentation.
You might think you’re all set after running one or two successful A/B tests, but you should always try to improve your digital marketing and promotional campaign efforts.
If you are constantly testing and improving your methods, then you will find they continue to get better. Not to mention, you’ll find new ways to stay ahead of the competition.
