Search rule experiment statistics

By understanding how the server collects and calculates statistics for the test elements in your search rule experiments, you can effectively evaluate the results of the experiment.

Here is an example of the statistical data that is provided for a search rule experiment:

Statistics for a search rule experiment

The following table describes the statistical data:

Statistic Description
Total number of unique customers The number of individual customers who participated in the experiment.
Unique Customers The number of individual customers who were assigned the path and reached the test element.
Clicks The number of times any link on the first page 1 of search results was clicked, which indicates that the search results had relevance.

This does not necessarily mean that the customer clicked a link for a product that is directly related to the search rule action. For example, if your search rule action is to boost Brand A products in the search results, the customer does not need to click a Brand A product to increase the Clicks count.

If the customer clicks a featured product that is displayed on the search results page, this click is not counted.

Click Orders The number of orders that are placed by customers who were influenced by the search results.

For the order to count in the statistics, the customer must add any item that is listed in the first page1 of search results to the shopping cart from the product display page that displays right after the customer clicks a link on the search results page.

Click Revenue For orders that counted as Click orders, the total value of all those orders.

For example, if a path has a count of 10 Click Orders, and each order totaled $10, the Click Revenue value is $100. This value is for the entire order, not just for the specific items in the order that were listed on the first page1 of search results.

1. By default, only the first page of search results is considered in click, order, and revenue statistics. However, a site administrator can change the default value of the SearchStatisticsResultPagesTrackingThreshold parameter to consider a different number of search result pages, for example, the first three pages.

You can view these statistics while the experiment is running and after it ends. See Viewing the statistics of a search rule experiment.

Enabling the collection of revenue statistics

To collect revenue statistics, a site administrator must enable the following statistical counters before your experiment starts:
  • ExperimentEvaluationEventListener
  • OrderSubmissionEvent

Frequency of statistical updates

The following table identifies how often statistics are updated in the Statistics tab while an experiment is running.

In the Clicks column The server gathers and displays these numbers using the SaveMarketingStatistics scheduled job. By default, this job runs every 15 minutes.

A site administrator can change the frequency of this job; see Scheduling the SaveMarketingStatistics job.

In the Click Orders and Click Revenue columns The server gathers and displays these numbers using the RaiseECEvent scheduled job. By default, this job runs every 5 minutes.

A site administrator can change the frequency of this job.

Experiment statistics in a staging environment

Search rule statistics are captured on the production server. If you want to view experiment statistics on a staging server for analysis, a system administrator can transfer the statistical data from the production server to the staging server by running the ExportStats and ImportStats commands. When you run the commands, specify the DMEXPSTATS and DMELESTATS tables.