Configuring an A/B test
To configure an A/B test, you specify the recipient sample sizes, content variations for each test split, test sequence, and the test evaluation criteria.
About this task
Before you can configure an A/B test, you must enable the mailing for A/B testing. If the mailing is already configured for A/B testing, a summary of the configuration settings displays in the Mailing Components section of the summary mailing tab.
If a document used in the A/B test includes constant personalization fields, you must define values for the fields after you save the A/B test configuration. Typically, you define values for constant personalization fields after you reference a document that contains constants with a mailing. However, when you enable a mailing for A/B testing, you can associate documents with the mailing only through the test splits for the A/B test. Therefore, you cannot define values for constant personalization fields until you save the A/B test configuration.
If the evaluation of recipient responses results in a tie between splits, the system declares the first split in the A/B test configuration as the winner.
Use the following steps to configure the A/B test or to update the current A/B test configuration.
Procedure
- On the summary mailing tab, click Configure to open the A/B Test Configuration window.
- Specify the size of the recipient sample for the test versions
of the email.
Each test split sends a version of the email to a portion of the overall recipient list. You specify the portion as a percentage of the total recipient list. The system randomly selects recipients from the list to add to each test split. Recipients in different splits see different versions of the email message.
- Select the email design or content variations to test.
For each variation, add a test. A test is equivalent to a test split.
You can add between two and five test splits to an A/B test.
For each test split, specify the following.
- Test name Each test must have a unique name.
- Document Select the document that contains
the email design or content that the system sends to the test split.
You can select only one document in a particular split, but you can
select different documents for each split in the A/B test.
Select a document that provides only HTML content. Text-only email documents do not support A/B testing.
- Multi-content If the document includes zones that contain multiple content elements, click Edit to select which content variant to display in the test split.
Note: Content that is controlled by personalization rules is not available for selection in an A/B test and does not appear in the Multi-content field.Preview the email for each split to see an example of the message that is sent to recipients in the split during the A/B test.
- Specify the split evaluation order. Click
to change the sequence. Move a split
up to run earlier. If there is a tie between splits, the first split to run is declared the winner. - Specify how long the A/B test must run by indicating when
the system evaluates test results. The system evaluates responses
received up to the time you specify.
You can specify a test duration or select a specific date and time when the system evaluates responses and transmits the winning version of the email.
Date and time options are set according to the system time for the server on which Campaign and Deliver are installed. The server time might be different from your local time. The current server time displays next to the date option on the A/B test Configuration page.
- Specify the criteria that the system must use to determine
which version of the email to send to the remainder of the mailing
list. This version of the email is considered the winner of the A/B
test.
You can select only one of the following.
- Maximum unique views
- Maximum unique clicks
- Minimum unique complaints
Results
When you save the A/B test configuration, a summary of the configuration displays in the Mailing Components section of the summary mailing tab. You can update the evaluation time and evaluation criteria at any time up to when the system begins to send the winning email.
The Mailing Components section displays a breakdown of the number of recipients in each test split. If you did not reference a recipient list to the mailing, the number of recipients in each split is zero. However, the portion of the total list that you selected for each split displays as a percentage.