A/B test mailings

You can use recipient responses to determine the most effective email design and message content by sending the mailing as an A/B test. In Deliver, you can configure and conduct an A/B test and automatically send the most effective version of an HTML email message to your audience.

Deliver does not support sending text-only email as an A/B test. For A/B testing, specify HTML only as the content type to send.

When you send a mailing as an A/B test, the system sends different email messages, or different versions of the same message, to between two and five sample groups of recipients. The sample groups are selected at random from the recipient list that is referenced by the mailing. You control the sample size, the number and type of message variations, the evaluation criteria, and how long the system waits to evaluate the responses. In the test, you can compare how recipients respond to various combinations of different message layouts, content variations, or subject lines.

You can start the A/B test immediately or you can schedule the test to start later. The system then sends the different email versions and collects recipient responses. After the system collects responses for the period that is specified in the test configuration, it evaluates recipient responses to each email version.

The system determines which version of the email best meets the specified success criteria and automatically sends that version to the remainder of the mailing list. The version that is sent to the rest of the list is called the "winner" of the A/B test.

Deliver provides the A/B Testing Performance Report to present evaluation results and to identify the winning version. Optionally, you can generate a Deliverability Report for each test version and for the test winner.

Conducting an A/B test requires that you complete the following steps, in order.

  1. Enable the mailing for A/B testing.
  2. Configure the A/B test.
  3. Start the A/B test and observe the results.
Note: Content elements and email subject lines that are associated with personalization rules are not eligible for comparison in an A/B test. Do not configure personalization rules in zones or subject lines that contain content that you want to include in an A/B test.