How many recipients to include in an A/B test

A/B testing sends different email content or design variations to different portions of the recipient list for the mailing that you enable for A/B testing. Each of the selected portions of the list is called a test split. The system constructs each test split by selecting recipients at random from the recipient list.

You specify test splits as a percentage of the total list and assign each split a unique name. All splits are the same relative size. By default, the system assigns 10% of the recipient list to each split. You can change the split size, but all splits are of the same size.

Recipients are assigned to only one split at a time. If the system selects a recipient for a test split, the recipient receives only the version of the message that is sent to that split. After the test and evaluation phases of the A/B test, the system sends the winning version of the message to the remainder of the original mailing list. For example, when you conduct an A/B test with three 10% splits, 80% of the recipient list receives the winning email variation.

To better understand how many individuals see the winning version of the message, consider the split distribution.

• Variation A goes to 10% of the entire mailing list.
• Variation B goes to a different 10% of the list.
• Variation C goes to yet another 10% of the list.

Variation C is declared the winner. So, according to the test configuration, the remaining 70% of the list also receives Variation C.

As a result of the A/B test, Variation C is delivered to 10% of the original mailing list as part of the initial test. Variation C is also delivered to the remaining 70% of the list as part of the final mailing. The total result is 10% + 70% = 80%. According to the test design, 80% of the individuals in original mailing list receive Variation C as the best version of the message.

To determine the appropriate split size for an A/B test, consider the total size of the recipient list and how many variations are being tested. If you select too small a split size, you might not have a representative sample for each variation. If you select too large a split size, you might send a non-winning content variation to a larger portion of the recipient list than is desirable.

When you save the A/B test configuration, the summary mailing tab displays the number, name, and size of the test splits. If a recipient list is referenced to the mailing, the mailing tab lists the exact number of recipients in the list and in each test split. You can configure test splits before you reference the list to the mailing, but the specific number of recipients for each test does not display on the summary mailing tab. Only the percentage of the list for each split is listed until you reference a recipient list to the mailing.