Getting Started with A/B Testing Campaigns

A/B Testing Campaigns allow you to test different versions of a single campaign to see how small changes can have a big impact on your results. Choose what you want to test, like the subject line or content, and compare results to find out what works and what doesn’t.

How A/B Testing Campaigns Work

Set Up the A/B Testing Campaign

You’ll choose a single variable type—Subject Line, From Name, Content, or Send Time—and create up to three variations. We’ll combine each variation with your other content to create combinations that we’ll send to different sets of subscribers, so no subscriber receives more than one combination of your campaign.

Choose Winner Criteria

Send the combinations to all your recipients at once if you have a small list, or if you’re testing Send Time. With other variables or a larger list, you can choose to send to a percentage of your recipients to test combination results, and then automatically or manually send the winning combination to your remaining recipients.

To choose the winner, use one of these options.

  • Automatic: Open or Click Rate
    Use these options if you want MailChimp to send the winning campaign to your remaining recipients after a set amount of time, based on the highest open or click rate.

Graphic showing how two combinations are sent and the winner is picked based on the higher click rate.

  • Manual: Report Statistics
    Use this option if you want to choose the winner based on reporting data or other factors that you find to be the most valuable. For example, if your campaign combinations sell different products, you may want to send the combination that has the highest number of sales even if it has a lower open or click rate.

Definitions

When we talk about A/B Testing Campaigns, we use some terminology that’s a little different from how we talk about other tools and tasks in MailChimp.

Variable
The element of your campaign that you want to test. With an A/B Testing Campaign, you can test one of four variables: subject line, from name, content, and send time. Each instance of the variable is called a variation.

Combination
A campaign created with a testing variable and your other content. To test three different From names, we'll create three different combinations of your campaign.

Test
The period of time after the combinations are sent, during which we compare the results. The test phase provides data that will be used to determine the campaign’s winning combination.

Winner
The campaign that performs the best. This can be chosen automatically based on open or click rate, or you can manually choose the winner based on the reporting data you find the most valuable.

Variables You Can Test

Subject Lines
Try different phrasing or sales offers to see what gets the most attention.

From Name
See if your subscribers are more responsive to emails coming from a person’s name or from the name of your company or organization. You’ll provide the From name and From email address you want to use for each combination.

Content
Create different versions of your content to see what gets a better response. Use this variable to test small content changes or completely different templates.

After your A/B Testing Campaign is sent, use the campaign report to compare the performance of specific links. For example, if you have the same link in two different campaign combinations, the campaign report will tell you in which combination that link performed the best.

Send Time
Learn when your subscribers are most likely to open your campaigns. This option tests specific days and times, so you won't be able to send a winning campaign to the remainder of your subscribers. Use this data to inform when to send or schedule future campaigns.

A/B Testing Campaign Ideas

Here are some common ways MailChimp users learn from A/B Testing Campaigns.

  • What day of the week gets better open rates?
  • Does a subject line with an incentive or a teaser work best?
  • Does including your company name in your subject line increase engagement?
  • Is it better to use your name as the From name, or your company’s name?
  • Does the time of day a campaign is sent affect the click rate?
  • Are subscribers more likely to click a linked image or linked text?
  • Do subscribers prefer a campaign that contains a GIF or one with static images?

Was this article helpful?
What can we do to improve articles like this?