Index synchronization and delta updates in WebSphere Commerce Search

The WebSphere Commerce Search index can typically fall out of synchronization with the latest production data over time. To maintain accuracy of the search result data, reindexing must be performed at a certain time during normal business operation. There are three forms of search reindexing, and depending on your business requirements, there are costs and benefits associated with each of the following options: Full reindexing, delta reindexing, and emergency updates.

Consider the following scenarios when you select a reindexing option:

Full reindexing: Performing a full reindexing rebuilds the entire search index. After the indexing completes, the search engine automatically uses the updated index. Full reindexing is effective when many updates are applied over time and the index starts to degrade in performance due to fragmentation. However, performing a full reindexing can take considerable time and therefore is only recommended to be performed over an extended interval, for example weekly.
Delta reindexing: Performing a delta reindexing performs only incremental updates on the existing operational search index. Delta reindexing is effective when the amount of changed data is manageable and small enough to be applied as a batch. Typically, delta reindexing is performed at regular intervals in a staging environment. After the changes are ready, it is published from the staging environment over to production through replication, thus minimizing the business impact. However, there might be business scenarios where a delta reindex might be too complicated, and therefore a full reindexing is preferred.; To perform a delta reindexing, set the fullbuild flag to false when preprocessing and building the search index. Otherwise, the default value is true, which performs a full preprocess and index build.

Preprocessing and index building utilities

The two steps to build the index are data preprocessing and index building. The following utilities perform the steps:

The preprocess utility extracts and flattens WebSphere Commerce data and then outputs the data into a set of temporary tables inside the WebSphere Commerce database. The data in the temporary tables is then used by the index building utility to populate the data into search indexes by using the Data Import Handler (DIH). When there are multiple indexes, for example, each language with its own separate index, the index is built multiple times.

For more information, see Preprocessing and building the search index.

Events that trigger search reindexes

The search index is triggered for reindexing when certain actions take place within WebSphere Commerce. Reindexing is required to avoid working with stale data, ensuring that the content you are working with or previewing is up-to-date.

In the Management Center, with workspaces enabled:

Task group commit
Store preview
Show facets (catalog entry index)
UpdateSearchIndex scheduler job
Note: This scheduler task triggers reindexing only on the base index. It skips workspace indexes to prevent unnecessary reindexing that is caused by uncommitted changes.

In the Management Center, with workspaces disabled:

Store preview
Show facets (catalog entry index)
UpdateSearchIndex scheduler job

In the Aurora starter store:

UpdateSearchIndex scheduler job.
Note: This scheduler job detects changes that are made from the authoring server by using the TI_DELTA_CATENTRY and TI_DELTA_CATGROUP temporary database tables. That is, after the quick publish task is approved, the runtime automatically publishes the search change history to the temporary tables in both the staging and production databases. It then starts a reindexing for emergency fixes from the production server.

Using utilities:

The di-preprocess utility when run with the -workspace workspaceId parameter.
The di-buildindex utility when run with the -workspace workspaceId parameter.

When to perform full search index builds

The WebSphere Commerce Search index is automatically built when certain business tasks are performed, as outlined in Common business tasks and their impact to the WebSphere Commerce Search index. In several cases, common business tasks result in delta index builds that do not pose a significant risk to production system performance. However, performing several delta index builds without occasional full index builds might result in the search index gradually degrading over time due to fragmentation. To avoid this issue, performing full search index builds when possible ensures that the search index performs well over time.

When Lucene receives a delete request, it does not delete entries from the index, but instead marks them for deletion and adds updated records to the end of the index. This results in the catalog unevenly spreading out across different segment data files in the search index, and might result in increased search response times. If you have a dedicated indexing server, consider scheduling a full search index build that runs in the background once per month, so that the deleted entries are flushed out, and to optimize the data.

Scheduler jobs

WebSphere Commerce Search contains scheduler jobs to trigger for index synchronization.

For more information, see Scheduling WebSphere Commerce Search jobs.

Search index approaches

Although catalog entries are mentioned in index synchronization and delta updates in WebSphere Commerce search, the same approach applies for the other search indexes or BODs such as ChangeCatalogGroup.

Search index optimization

There are several search optimization hints and tips to consider when administering WebSphere Commerce search.

For more information, see Performing search index optimization.