Preparation and design for an A/B test

Running a meaningful A/B test requires careful planning and specific comparisons between the versions of the message that you are testing. You must evaluate several characteristics that affect the test outcome and usefulness.

Before you run an A/B test, consider the following issues.

  • The number and type of variations to test. You can compare email layouts, content designs, subject lines, or a combination of all.
  • The size of the recipient sample for the different versions of the email message.
  • How long to run the test. This decision affects how many recipients can respond in time to participate in the test.
  • Criteria that the system uses to evaluate recipient responses.

You can compare results for different email layouts, different content elements within the same email layout, or a combination of different layouts and different content. You can also test different email subject lines. You can test between two and five variations in a single A/B test.

Each variation that you add to the A/B test requires that you allocate a portion of the recipient list to receive and respond to it. The individual portions of the list that receive a different version of the message are often called test splits. You can vary the size of the test splits. Consider how much of the recipient list you select for the test splits. Recipients in the test splits do not receive the winning version that the system sends to the rest of the list.

You control how long to run the test by specifying a test duration or by specifying the exact evaluation date and time. Consider how much time is necessary to create an accurate view of recipient opinion about the email variants.

You can choose between several options for evaluation criteria when you configure how to evaluate recipient responses to determine the winning variation. You must select one of the options that are provided on the A/B Test Configuration page.