A/B testing of treatment rules

A/B testing, also known as 'split testing' is the process of comparing two versions of same scenario with just one varying parameter. It brings scientific methodology to marketing and removes the guesswork.

In Interact, A/B testing is the test done for a treatment rule with all properties fixed, except one varying parameter. In version 12.1 and higher releases, this parameter is the combination of offer and/or offer attributes. In an A/B testing of a rule, each variation of offer and/or offer attributes represents one A/B testing branch within same audience distribution and same zone. One treatment rule can only have one A/B Testing defined. One A/B testing can have multiple branches defined depending on the marketers testing requirements. But un-necessary branches can degrade performance in the runtime process.

An A/B testing is executed in Runtime in the period of time defined by effective date and expiration date. Both dates are optional. If the effective date is not defined, then A/B testing is in effect immediately after deployment. If expiration date is not defined, then it is in effect forever.

In the period of A/B testing, one of the offers from A/B testing branches is randomly served to visitors. Once A/B testing is completed, i.e., after its expiration date, the branch which remains in execution is determined by the Selection mode specified by marketers in A/B testing definition. The following three selection modes are supported.

  • Default branch: User selects a branch from all branches in an A/B testing definition.
  • Auto selection: It is based on the history of acceptance data. Interact runtime automatically selects the branch that has the highest acceptance rate.
  • Use base: Base treatment rule is used so that arbitration logic is resumed as it was before the A/B testing.

In Interact runtime, all branches in an A/B testing are treated like treatment rules. The ultimate goal of an A/B testing is to find the treatment rule (combination of an offer and offer attributes), which has the best acceptance rate from visitors. The acceptance rate is a ratio of number of an offers being accepted to number of the offers being contacted. When a contact or an accepted event is posted for a treatment rule and if this treatment is the result of one of the branches of an A/B test, Interact runtime inserts or updates the data in the UACI_ABTestPerformance table in Runtime database. At the end of the A/B testing period, the contents in UACI_ABTestPerformance table is used to calculate the acceptance rates of all the branches belonging to the A/B testing. The best performing branch within all branches can then be determined by selecting the highest accept to contact ratio.