WebSphere Commerce search programming model, extension points, and customization tasks

This guide outlines the WebSphere Commerce search programming model, extension points, and how it can be customized. There are mandatory and optional areas to work with, depending on the customization that is being performed to your WebSphere Commerce search implementation.

Most customization flows involve indexing new data from either the database or a file, searching against the new indexed fields, displaying the indexed data in the storefront, and creating business rules by using the indexed data field. There are other advanced search customization areas that require some runtime customization where a custom search expression provider, search result filter, or other runtime search extension points might require customization.

Topic overview

This topic contains several sections of interest that are related to the WebSphere Commerce search programming model, extension points, and customization roadmap.

The following list helps summarize the key areas of interest:

Search index schema customization
- Local index
- Extension index
Search indexer customization
- Preprocessor and Data Import Handler (DIH)
- Dataload
Search runtime customization
User interface customization
- Management Center
- Storefront

These extension points are described in the following flow diagram:
WebSphere Commerce search architecture

The catalog entry customizable fields, field1 (Integer), field 3 (Decimal), and field5 (Text) from the CATENTRY table are used as an example to help clarify the customization process.

Search index schema customization

Adding new fields into the index schema requires modifying the schema.xml file to add the new index field. Typically, an existing index is updated to add new fields (local index). In other instances, it is recommended to extend the product index by creating an index as an extension of an existing index (extension index).

The following snippet is a sample index field for the catentry_id in the CatalogEntry index:


<field name="catentry_id" type="string" indexed="true" stored="true" required="true" multiValued="false"/>

WebSphere Commerce field types

In addition to native Solr field types, WebSphere Commerce introduces custom field types. For example:

wc_text: Tokenized text for searching.
wc_keywordText: Non-tokenized text for sorting and faceting.
wc_keywordTextLowerCase: Non-tokenized text for not case-sensitive sorting.
wc_textSpell: Spell checking for text fields.

Valid attributes for fields

The following list summarizes the valid attributes for fields;

name: (Required) The name for the field.
type: (Required) The name of a previously defined type from the <types> section.
indexed: Indicates whether the field should be indexed (searchable or sortable).
stored: Indicates whether the field should be retrievable.
compressed: Indicates whether the field should be stored by using gzip compression.; This applies only if the field type is compressible. For the standard field types, only TextField and StrField are compressible.
multiValued: Indicates whether the field can contain multiple values per document.
omitNorms: Indicates whether to omit the norms that are associated with this field. Setting to true disables length normalization and index-time boosting for the field, and saves some memory.; Only full-text fields or fields that require index-time boosting need norms.
termVectors: Indicates whether to store the term vector for a given field.; When using MoreLikeThis, fields that are used for similarity should be stored for best performance.

Note: When you work with an extension index, be aware of the following guidelines when you use the indexed and stored fields:

If an extension index is used as a filter index, where the field is always included in a filter query and the column value is irrelevant, set the stored value to false.
Otherwise, if the extension index always works to supplement meta information to the master core, such as inventory, set the stored value to true.
If facets are required by a field in the extension index, set the indexed value to true.
Setting the stored field as false might save index storage space.
If a column copies to a dest column, it is unnecessary to set stored as true.

WebSphere Commerce text field naming convention

The following naming convention is followed for text fields in WebSphere Commerce search:

fieldName: Tokenized and not case-sensitive.; For example, mfName.
fieldName_cs: Tokenized and case-sensitive.; For example, mfName_cs.
fieldName_ntk: Non-tokenized and not case-sensitive.; For example, mfName_ntk.
fieldName_ntk_cs: Non-tokenized and case-sensitive.; For example, catenttype_id_ntk_cs.

Note:

WebSphere Commerce search requires one search index per language. Whenever the search schema is changed and the index is regenerated, it must be performed for every language.
To view the customization under the workspace preview, any changes that are made to the base schema.xml file must also be applied to the workspace schema.xml file.

Local index

The index fields are defined in the fields section of the schema.xml file in the following directory:

WC_installdir\components\foundation\subcomponents\search\solr\home\template\CatalogEntry\conf\locale\locale

Note: The schema.xml file is locale-specific, as some data types use locale-specific tokenizers.

To customize the index schema, modify the schema.xml file. The following configuration is an example of adding the catalog entry customizable fields, field1, field3, and field5 to the CatalogEntry index:


<field name="catentry_field1" type="tint" indexed="true" stored="true" required="false" multiValued="false"/> 
<field name="catentry_field3" type="tfloat " indexed="true" stored="true" required="false" multiValued="false"/> 
<field name="catentry_field5" type="wc_text" indexed="true" stored="true" required="false" multiValued="false"/>

Extension index

The base index schema can be extended to suit your business needs. For example, to separate data into different indexes that are based on their refresh intervals.

For more information, see Extending the WebSphere Commerce search base index schema.

Indexing customization

The indexing process requires mapping data from an external source (database or file) to index fields to create index records or index documents. Depending on the complexity of the source data and index schema design, preprocessing the data might be required. Otherwise, indexing can be directly performed by using either the Data Import Handler (DIH) to index data from the database, or SolrJ APIs to index data from a file.

Preprocessor and Data Import Handler (DIH)

The preprocessing tasks are controlled by the wc-dataimport-preprocess XML files. The files contain table definitions, database schema metadata, and references to the Java classes used in the preprocessing steps. Those files are invoked by the di-preprocess utility, which extracts and flattens WebSphere Commerce data and then outputs the data into a set of temporary tables inside the WebSphere Commerce database. The data in the temporary tables is then used by the index building utility to populate the data into Solr indexes that use the Solr Data Import Handler (DIH).

The index building utility uses DIH to connect to the WebSphere Commerce database through a JDBC connection. It crawls the WebSphere Commerce tables, and then populates the Solr index. The JDBC configuration and crawling SQL statements are defined in the wc-data-config.xml configuration file.

For more information, see Customizing the preprocessor and Data Import Handler (DIH).

Dataload indexing

The default Dataload indexer generic mediator enables indexing indexer-ready data directly into WebSphere Commerce search. A new mediator can be created to support new data types.

For more information, see Extending the Dataload indexer mediator for WebSphere Commerce search.

WebSphere Commerce search runtime customization

The following areas of the WebSphere Commerce search run time can be customized:

Expression providers

An expression provider implements custom business logic for modifying the main search expression before it is sent to the search engine. Each expression provider can be separately registered to any search profile query section so that it can be reused for other search operations. That is, a search profile defines a list of search providers that are used for assembling the main expression for a search request.

For more information, see Creating a custom expression provider.

Result Filters

A result filter modifies the logical object (noun) before the search result is returned to the storefront. The filter has full access to the CatalogNavigationView noun. Therefore, more data can be inserted, updated, or removed from any parts of the noun and its noun parts, such as CatalogEntryView or CatalogGroupView. Each result filter can be separately registered to any search profile result section so that it can be reused for other search operations.

Note: Result filters must not contain any search engine-specific code.

For more information, see Creating a custom result filter.

Disabling default search expression providers and result filters

You can disable certain default expression providers and result filters to optimize the storefront flow and improve the overall response time for certain types of WebSphere Commerce search scenarios. For more information, see Disabling search expression providers and result filters in the search configuration file (wc-search.xml).

Query preprocessor

A query preprocessor modifies the native SolrQuery object right before it is sent to the Solr server for processing.

For more information, see Creating a custom query preprocessor.

Query postprocessor

A query postprocessor modifies the physical DataObject, SolrEntityContainerImpl, immediately after the QueryResponse is returned from the Solr server.

For more information, see Creating a custom query postprocessor.

Search profiles

WebSphere Commerce search uses search profiles to control the storefront search experience at a page-level. Search profiles group sets of search runtime parameters such as search index name, search index fields, expression providers and result filters, paging and sorting, and search feature configurations such as text highlighting, facets, and spelling correction. Search profiles are defined in the WebSphere Commerce search configuration file (wc-search.xml). For more information about the default search profiles, see WebSphere Commerce search configuration file (wc-search.xml) (WC EAR).

For more information, see Customizing search profiles.

Search mediator

A custom search mediator has access to both the logical noun object and the native Solr search result set object, SolrDocumentList.

Note: Search Mediator is always called upon regardless of which search profile used, as its configuration is defined in the wc-business-object-mediator.xml file and not in wc-search.xml. To modify the logical object based on a given search profile, use Result Filters. For more information, see Creating a custom result filter.

For more information, see Customizing search mediators.

Business object mediator

The business object mediator maps physical search index fields to their corresponding logical fields in the CatalogEntryView noun. Custom physical index fields can be mapped by using the UserData fields.

For more information, see Customizing business object mediators.

User interface customization

The following areas of the WebSphere Commerce user interface can be customized:

Management Center

Search rule actions and targets in the Management Center dynamically populate a list of catalog entry indexed properties. You must customize WebSphere Commerce search when you are adding new indexed catalog entry properties to search rule actions or targets.

Note: You cannot change the property name in search rules once saved. That is, filters of type property are read-only once saved.

The following screen captures show the default property selector in the Management Center:

To customize the property selector, follow the steps in Adding catalog entry properties to search rule actions or targets.

To migrate to the new property selector in the Management Center, follow the steps in Migrating catalog entry properties to search rule actions or targets.

Storefront

You can customize WebSphere Commerce search in the storefront to suit your business needs. You can customize WebSphere Commerce search according to your role within the WebSphere Commerce site.

For more information, see Customizing WebSphere Commerce search in the storefront.