A/B testing tab

A/B testing, also known 'split testing' is the process of comparing two versions of same scenario with just one varying parameter. It brings scientific methodology to marketing and removes the guesswork.

In Interact, A/B testing is the test done for a treatment rule with all its properties fixed, but only one parameter varies. This parameter is the combination of offer and offer attributes. In an A/B testing of a rule, each variation of offer and/or offer attributes represent one branch with specified audience distribution.

The testing period is defined with effective date and expiration date. Both dates are optional. If effective date is not defined, then A/B testing is in effect immediately after deployment. If expiration date is not defined, then it is in effect forever.

By default, A/B testing is disabled for a rule. To enable it, click , the A/B icon to the right of a rule. The Advanced Options slider displays. Perform the following steps.

  1. Select the Enabled checkbox. This adds base rule as first branch in Branch table.
  2. Under TestPeriod, provide the Effective and Expiration dates.
  3. Inthe Selectionmode list, select one of the following options.
    • Use Default: Select this option if you want the offer to be served from the default branch out of A/B test branches, regardless of the higher acceptance rate of any offer.
    • Use base: Select this option if you want the offer to be served from the treatment rule, regardless of the higher acceptance rate of any offer.
    • Auto Selection: Select this option if you want the offer to be served from one of the A/B test branches having highest acceptance rate. Acceptance rate is calculated as ratio of response count to contact count.

      Once "Auto Selection" or "Use Base" option is selected, then Default Branch column willbe hidden. If there is performance data in Acceptance Rate column, then branch information can be read only.

  4. ClickAdd Branch to add more branches in Branch table.
  5. Click Apply changes.

Branch Name is an unique identifier for a branch. It is a required field and must be unique within A/B testing. A branch can be enabled and disabled. Only enabled branches participate in A/B Testing at runtime. An A/B testing can be enabled or disabled. Only enabled A/B Testing is actually executed at runtime.

To save A/B testing data, click Apply Changes on Advanced Option slider.

A/B Testing data is saved to the database only when you click the “Save” or “Save and Exit” button on the upper right corner on Strategy rule list.

In the Strategy Rule list page, select the ShowA/B testing rules only checkbox to display the rules with A/B testing and acceptance rate column with ABT status and acceptance rate of each branch.

There can be four statuses for Acceptance Rate column for base rule including disabled, yet to start, in progress, and finished. In case, you select the "Use Default" or "Auto Selection" selection mode, a "check-mark" near the selected branch is displayed.

For any rule that has A/B testing data defined, there is "A/B Testing" icon located on the right. The orange color means it is active and grey color means it is disabled.

ABT Summary page displays the “ABT status” and “number of branches” for the A/B test.

To analyze A/B Testing results in case of "Auto Selection" mode, a column "ABTestBranchID" is added into audience specific detail contact history table (Example: UA_DTLContactHist table) in Campaign database to record the ID of the selected branch when A/B testing is engaged in arbitrating the relevant smart rules. The information in this column, together with other treatment fields, enables users to create reports on the performance on A/B testing branches for individual smart rules. The "ABTestBranchID" column is used in rule based performance query for retrieving the branch details and calculate the accept rate of branches.

One of the offers from A/B testing branches is served based on the random selection approach if A/B testing is define but not completed. Once A/B testing is completed, i.e. after its expiration date, then an offer from A/B testing branches is served as per the option selected under the Selection Mode list.

If Mode is "Auto Selection", then UACI_ABTestPerformance table of runtime database will be used for serving the offer.

When a contact or an acceptance event is posted for the treatment and if this treatment is the result of anA/B test, then it inserts or updates the data in the UACI_ABTestPerformance table of runtime database.

Note: The abTestBranchID column must be populated in dtlcontacthist table in order to display acceptance on ABT page. For this, ETL tables must be mapped on the previous version.
Note: If offer codes are used while posting an event, this event may be mistakenly counted into A/B testing even it is meant to be posted against a treatment from a different source, and vice versa. In this case, the report may be erroneous.

During offer arbitration, when a rule has A/B testing enabled but expired, the UACI_ABTestPerformance table calculates the acceptance rate of all the branches belonging tothe rule and the best performance branch is calculated based on their highest accept to contact ratio.

The UACI_ABTestPerformance table is queried during offer arbitration and data of this table is stored in the read cache. An offer is severed based on the data that existsin the read cache. The data of the read cache is updated after 60 minutes as default value of the JVM parameter com.unica.interact.abtPerformanceDataLoadFrequencyInMin is 60 minutes. You can change the data update time of the read cache by using the com.unica.interact.abtPerformanceDataLoadFrequencyInMin JVM configuration parameter.

If you have the same acceptance rate for multiple branches, then the offer of that branch is selected during offer arbitration, which includes the current updateTime value.