Search rule experiments
Search rule experiments provide a way to test the effectiveness of specific search rules running on your storefront. By setting up a search rule experiment, you can run alternative paths in a single search rule and compare the results of each path. You can also test whether a specific search rule is producing better results than having no search rule at all. For each path in the experiment, you can see statistics for unique customers, clicks, orders, and revenue.
Example 1
A store sells electronics, including digital cameras. The marketing manager wants to experiment with two search rule actions; one boosts Brand A digital cameras in the search results and the other boosts Brand B digital cameras. To see which set of search results generates more clicks, orders, and revenue for the store, he sets up a search rule as an experiment with two paths:
When this experiment runs on the storefront, and customers search the site for "digital camera":
- 1 For 50% of customers, path 1 is applied. Path 1 boosts Brand A digital cameras in the search results using the Change Search Result Order action. In other words, customers searching for digital cameras see Brand A cameras closer to the top of the search results.
- 2 For the other 50% of customers, path 2 is applied. Path 2 boosts Brand B digital cameras in the search results instead, also using the Change Search Result Order action.
To see which path is generating more clicks, orders, and revenue for the store, the marketing manager reviews the statistics for the experiment after a week has passed:
Based on these statistics, the search action on path 1 that boosts Brand A digital cameras is more effective. In response to these statistics, the marketing manager selects path 1 as the "winning" path and uses the search rule action that boosts Brand A digital cameras on the storefront going forward.
Example 2
A marketing manager wants to see if she can increase sales by listing a specific set of trendy shoes at the top of the search results whenever young female customers search the site for "shoes." She wants to be sure that her search rule is having a positive effect on sales, compared to having no search rule at all. To find out, she creates the following search rule experiment with an empty control path (Path 1) that does not apply a search rule action to the search results:
When this experiment runs on the storefront, and young female customers search the site for "shoes":
- 1 50% of these customers see search results that have no search rule action applied.
- 2 The other 50% of these customers see the trendy shoes that the marketing manager has chosen to display at the top of the search results.
The marketing manager checks the Statistics tab for the experiment after a few days, and the results are shown here:
Based on these statistics, the marketing manager can see that her selection of shoes for young women is more relevant to this customer segment, and that the search rule action is helping to increase orders and revenue. She selects Path 2 as the "winning" path and continues to use this search rule action for the remainder of the spring shoe season.
Guidelines for creating search rule experiments
Following these guidelines for creating search rule experiments:- You can turn any search rule into an experiment by dragging the
Experiment element into
the rule to the left of the target or action element that you want
to experiment with. - You should have only one search rule experiment active on your storefront at a time; however that search rule experiment can have two or more paths to test different scenarios. Experiment paths can also use Branch elements to add complexity.
- You can use different actions on each path of an experiment. For example, you can have a Specify Top Search Results action on one path and a Change Search Result Order action on another path. You can also use more than one action on a single path.
- Including targets in your experiment is optional. The target can be either to the right or to the left of the experiment element. You can use multiple targets of different types in an experiment. For a list of available targets, see Targets in search rules.
- Avoid leaving search rule experiments running indefinitely because they use resources that might affect performance. When you decide on a winning path in the experiment, create a new search rule from the winning path according to the instructions in Selecting the winning path for a search rule experiment.
In an extended site model, the statistics
reflect the results in the current store. If the search rule is defined
in the asset store, then each store accumulates its individual statistics.
In this way, you can evaluate the effectiveness of the same experiment
in different stores. Search rule experiments created in the asset
store can be used in an extended site.