Create an A/B Testing Campaign

This feature requires Author user level or higher.

Each time you create a campaign, there are many ways to present your content, but you may wonder which way is best. Use an A/B Testing Campaign to test how small changes to your campaigns impact your results.

Before you create an A/B Testing Campaign, familiarize yourself with how A/B Testing Campaigns work and the terminology.

Below, you'll learn how to create and edit A/B Testing Campaigns.

Best Practices

Include 5,000 Subscribers per Combination

For the most useful data, each combination in your A/B Testing Campaign should go to a large number of your subscribers. 5,000 subscribers per combination is a good general rule. If the size of your list or segment doesn't allow for 5,000 subscribers per combination, we strongly recommend you test your entire list. In these cases, you can use those campaign results to inform future campaign content decisions.

Set a Four-Hour Test Time

Allow at least four hours of test time when you send an A/B Testing Campaign to a percentage of your list. Result accuracy increases with each additional hour that you wait. The first couple of hours of results aren't always representative of your testing segment, and could result in sending the less effective campaign to the rest of your list.

Most subscribers don't check their email right as it comes in, and spam filters and incoming server volume may slow down email delivery speed. Because of these factors, set a test time of at least four hours to give your subscribers plenty of time to receive and engage with your campaign.

Test Content by Click Rate

When you test campaign content, choose your winner by the best click rate. Opens occur before a subscriber sees the content, so open rate isn't an accurate indicator of how engaging your content is. If a subscriber clicks a link in your content, however, there is a high likelihood that they read your campaign.

Create an A/B Testing Campaign

A/B Testing Campaigns are built in a series of steps, like regular campaigns. If you haven't started building an A/B Testing Campaign yet, start at the beginning of these instructions. If you already started and saved your progress, skip to the next step in the process.

To create a new A/B Testing Campaign, follow these steps.

  1. Navigate to the Campaigns page.
  2. Click the drop-down menu next to Create Campaign, and choose A/B Testing Campaign.
    Create Campaign drop-down menu, with cursor clicking A/B Testing Campaign.

Recipients Step

On the first step of the Campaign Builder, choose the list or segment you want to send your campaign to. To send to a segment, configure a new segment on this screen, or choose a saved segment.

Variables Step

On the Variables step, you'll configure the elements you want to test.

  1. Click the Plus icon for the variable you want to test.
    Image displaying variate choices
  2. Click the Plus or Minus icons to add or remove additional variations of the variable.
    Cursor hovers over plus icon to choose 3 subject lines.
    You can test one variable with up to three variations. The number of variations equals the number of combinations, as shown in the Summary.
    Screen of 3 subject lines totalling up to 3 combinations.
    For example, to test three subject lines, we'll generate three combinations of your campaign.
  3. Under How would you like to test the combinations, choose the percentage of your list you want to send your testing combinations to.
    Screen of the Distribute the combinations across all recipients option selected.
    • Distribute the combinations across all recipients
      Sends to your entire list. We recommend this option for lists that aren't large enough to have 5,000 subscribers per combination. If you choose Send time as your variable, you must choose this option.
    • Test on a percentage of recipients to send a winner
      Sends combinations to the testing portion of your list, and sends the winning combination to the remainder of the list.

    If you selected to test based on a percentage, click and drag the slider to set the percentage of your list to send your test to.
    Screen of slider choosing an 80% test.

    Use the Test segment and Winning segment portion of the Summary to see how many of your subscribers will receive each combination.
    Screen of Summary table, showing 3 total combinations and the total recipents for the campaign and each combination.

    Choose whether to determine a winner based on open rate, click rate, or manually by other report statistics. Then, choose how long you want the test to run. The winning combination is chosen at the end of the selected test time and sent to your remaining recipients. During the test phase, you can convert to manual selection to choose the winner from the campaign report.

Setup Step

You'll configure settings for the campaign on the Setup step.

  1. Type in the test name, Email subject, and From email address and name. If you are testing Subject line or the From name, you'll see fields for each variation you want to test. Send time variations will also be set on this step.
    Screen of three different subject line options.
  2. Check the boxes next to any additional settings or tracking options to include.

Content Step

The Content step is where you’ll add content for your campaign. What you see on this step will vary if you choose to test content.

Test Subject Line, From Name, or Send Time

  1. Choose your  layout on the Template step of the Content Builder. On the Design step of the Content Builder,  design and  test your campaign. When everything is how you want, click  Confirm before sending.

Test Content

The Content setup page shows a preview for each variation of your campaign content. To add content to each variation, follow these steps.

  1. Click Add Content for a variation of your campaign.
    Screen of content setup page with cursor hovering over to add content to the first variation.
  2. Choose your  layout on the Template step of the Content Builder.
  3. On the Design step of the Content Builder, design and test your campaign. When everything is how you want, click Next.
  4. On the Description step of the Content Builder, type in text to describe the variation of the content. Your subscribers won't see this description, but it will help you remember the differences between each variation.
    Screen of Content description step, with Administration Building Photo to describe the content.
  5. Click Continue to content to return to your content variations.
  6. Repeat steps 1-5 for each content variation. On the Template step, use a different template or select a variation you already designed from the Previous Variations tab.
    Screen of Previous Variations tab with one variation option.
  7. When everything is how you want, click Next.

 

Confirm Step

The Confirm step is where you’ll review the combinations that make up your A/B Testing Campaign.

  1. Review the Pre-Delivery Checklist. Click Resolve or Edit to make changes to any section of the campaign.
  2. Send your campaign now, or schedule it to go out later.
    Screen of Schedule and Send buttons.

Manually Choose a Winner

If you chose Manual Selection from the drop-down menu on the Variable step of the Campaign Builder, you'll be able to view the campaign report after tests are sent, and choose a winner.
Screen of Manual Selection choose from the What metric should determine the winner drop-down menu.

You'll also receive an email to remind you to select your winner.

The option to manually choose a winner allows you to control the combination that is sent to your remaining subscribers and when.

To manually choose a winner, follow these steps.

  1. Navigate to the Reports page.
  2. Click View report for the A/B Testing Campaign you want to work with.
  3. An alert will tell you how long into the testing phase you are. If it is less than four hours, we recommend that you wait.
    Feedback block showing information about how long the test has run and the recommendation to wait 4 hours before choosing a winner.
  4. Review the report for each combination. Under Finalize your test, click the radio button next to the combination you want to send to your remaining recipients.
    Screen of cursor choosing an option from the combination results table.
  5. Click Send To Remaining Recipients.
    Cursor clicks the Send To Remaining Recipients button.
  6. Click Send Now in the confirmation pop-up modal to send the winning campaign.

Convert to Manual Selection

After you send your test with the choice to have the winner chosen by open or click rate, you may decide that you want to manually choose the winning combination instead.

To convert to manual selection after you sent your campaign, follow these steps.

  1. Navigate to the Reports page.
  2. Click View Report for the A/B Testing Campaign you want to work with.
  3. Click Convert to manual selection.
    Feedback block showing the cursor clicking the option to convert to manual selection.
  4. In the pop-up modal, type CONVERT in all caps and click Convert.
    Screen of modal with CONVERT typed in the field and the cursor clicking Convert.
  5. When you’re ready, choose a combination to immediately send to your remaining subscribers.

Was this article helpful?
What can we do to improve articles like this?