Index synchronization and delta updates in WebSphere Commerce search

The WebSphere Commerce search index can typically fall out of synchronization with the latest production data over time. To maintain accuracy of the search result data, reindexing must be performed at a certain time during normal business operation. There are three forms of search reindexing, and depending on your business requirements, there are costs and benefits associated with each of the following options: Full reindexing, delta reindexing, and emergency updates.

Consider the following scenarios when selecting a reindexing option:

Full reindexing: Performing a full reindexing rebuilds the entire search index. Once the indexing completes, the search engine automatically uses the updated index. Full reindexing is effective when many updates have been applied over time and the index starts to degrade in performance due to fragmentation. However, performing a full reindexing can take considerable time and therefore is only recommended to be performed over an extended interval, for example on a weekly basis.
Note: All changes performed in Workspaces require a full reindexing when the workspace is committed into the base schema.
Delta reindexing: Performing a delta reindexing performs only incremental updates on the existing operational search index. Delta reindexing is effective when the amount of changed data is manageable and small enough to be applied as a batch. Typically, delta reindexing is performed at regular intervals in a staging environment. Once the changes are ready, it is published from the staging environment over to production through replication, thus minimizing the business impact. However, there might be business scenarios where a delta reindex might be too complicated, and therefore a full reindexing is preferred.; To perform a delta reindexing, set the fullbuild flag to false when preprocessing and building the search index. Otherwise, the default value is true, which performs a full preprocess and index build.

Preprocessing and index building utilities

The two steps to build the index are: data preprocessing and index building. The following utilities perform the steps:

The preprocess utility extracts and flattens WebSphere Commerce data and then outputs the data into a set of temporary tables inside the WebSphere Commerce database. The data in the temporary tables is then used by the index building utility to populate the data into search indexes using the Data Import Handler (DIH). When there are multiple indexes, for example, each language using its own separate index, the index is built multiple times.

For more information, see Preprocessing and building the search index.

Events that trigger search re-indexes

The search index is triggered for re-indexing when certain actions take place within WebSphere Commerce. Re-indexing is required to avoid working with stale data, ensuring that the content you are working with or previewing is currently up-to-date.

In the Management Center, with workspaces enabled:

Task group commit
Store preview
Show facets (catalog entry index)
UpdateSearchIndex scheduler job
Note: This scheduler task triggers re-indexing only on the base index. It skips workspace indexes to prevent unnecessary re-indexing caused by uncommitted changes.

In the Management Center, with workspaces disabled:

Store preview
Show facets (catalog entry index)
UpdateSearchIndex scheduler job

In the Aurora starter store:

UpdateSearchIndex scheduler job.
Note: This scheduler job detects changes made from the authoring server using the TI_DELTA_CATENTRY and TI_DELTA_CATGROUP temporary database tables. That is, after the quick publish task is approved, the runtime automatically publishes the search change history to the temporary tables in both the staging and production databases. It then launches a re-indexing for emergency fixes from the production server.

Using utilities:

The di-preprocess utility when run with the -workspace workspaceId parameter.
The di-buildindex utility when run with the -workspace workspaceId parameter.

When to perform full search index builds

The WebSphere Commerce search index is automatically built when certain business tasks are performed, as outlined in ../refs/rsdsearchindexhints.html. In several cases, common business tasks result in delta index builds that do not pose a significant risk to production system performance. However, performing several delta index builds without occasional full index builds might result in the search index gradually degrading over time due to fragmentation. To avoid this issue, performing full search index builds when possible ensures that the search index performs well over time.

When Lucene receives a delete request, it does not delete entries from the index, but instead marks them for deletion and adds updated records to the end of the index. This results in the catalog unevenly spreading out across different segment data files in the search index, and might result in increased search response times. If you have a dedicated indexing server, consider scheduling a full search index build that runs in the background approximately once per month, so that the deleted entries are flushed out, and to optimize the data.

Scheduler jobs

WebSphere Commerce search contains scheduler jobs to trigger for index synchronization.

For more information, see Scheduling WebSphere Commerce search jobs.

Search index approaches

Although catalog entries are specifically mentioned in index synchronization and delta updates in WebSphere Commerce search, the same approach applies for the other search indexes or BODs such as ChangeCatalogGroup.

Performing search index optimization

There are several search optimization hints and tips to consider when administering WebSphere Commerce search.

For more information, see Performing search index optimization .