Migrating custom index fields and indexing scripts from BOD-based search to REST-based search

Any customization that is made to any of the index schemas, or to the preprocessing and indexing scripts can be reapplied when you migrate from BOD-based search to REST-based search.

Search in HCL Commerce Version 9 uses Solr 7.3.1, while BOD-based search used previous version of Solr. Therefore, your search index must be rebuilt and customized fields must be reimplemented. HCL Commerce Version 9 Search also uses RESTful API to build the search index. For more information, see Building the search index.

Procedure

Migrate your index schema customizations. In HCL Commerce Version 9, the catalog entry fieldType definitions use two templates:
- A non-customizable template: search-config/src/main/resources/managed-solr/config/v3/common/schema-field-types.xml.
  Note: When the first index is created, this XML file is copied to the resources/search/index/managed-solr/config/v3/common directory. After the index creation, other indexes share this fieldType definition.
- A customizable template: search-config-ext/src/main/resources/index/managed-solr/config/v3/common/x-schema-field-types.xml.
  Note: When the first index is created, this XML file is copied to resources/search/index/managed-solr/config/v3-core-extension/common directory. After the index creation, other indexes share this customizable fieldType definition.

Migrate your custom schema fieldType by using the following example.

If you created a new field type x_textSpell for the fr_FR locale in a previous HCL Commerce version, you updated the schema.xml file, which was located under core configuration directory specific to the fr_FR locale. In HCL Commerce Version 9, you must add the field type x_textSpell_fr to the file x-schema-field-types.xml as follows.

<!-- Spell checking text field -->
<fieldType omitNorms="true" positionIncrementGap="100" class="solr.TextField" name="x_textSpell_fr">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
<filter class="solr.WordDelimiterFilterFactory" preserveOriginal="0" splitOnNumerics="1" splitOnCaseChange="1" 
    catenateAll="0" catenateNumbers="0" catenateWords="0" generateNumberParts="1" generateWordParts="1"/>
<filter class="solr.ShingleFilterFactory" fillerToken="" tokenSeparator=" " maxShingleSize="3" minShingleSize="2" 
    outputUnigrams="true"/>
<filter class="solr.PatternReplaceFilterFactory" replace="all" replacement=" " pattern="\s{2,}"/>
<filter class="solr.TrimFilterFactory"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>

<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StopFilterFactory" words="${stopwords_fr:../../../v3/common/stopwords.txt}" ignoreCase="true"/>
</analyzer>
</fieldType>

After the new x_textSpell_fr field type is defined in the x-schema-field-types.xml file, define an entry into your x-schema.xml file by using the following example.
```
<field name="spellCheck" type="x_textSpell_fr" indexed="true" stored="false" multiValued="true" />
```
Notes:
- If you want to support all languages, you must create a field type for every language in the x-schema-field-types.xml, then add the following field definition into x-schema.xml file.
```
<field name="spellCheck" type="x_textSpell_${lang:en}" indexed="true" stored="false" multiValued="true" />
```
Migrate your customized stop words.
In previous HCL Commerce versions, you could change the stopwords.txt file for each core configuration directory. For example, if you customized stopwords.txt for the English language, you need only to change the stopwords.txt file under the en_US core. When you migrate to HCL Commerce Version 9, the index schemas are shared among multiple languages. In order to customize stopwords.txt for a specific language, HCL Commerce Version 9 requires you point to the file in the SRCHCONFEXT database table. For more information about customizing the stopwords.txt, see Customizing the stopwords.txt file. Run the following SQL command to put a stopwords.txt into the config field:
```
Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt' 
where indextype=’CatalogEntry’ and indexscope=’$MASTERCATALOGID’
```
When the HCL Commerce Search server is restarted, the following contents are added into x-core.properties under the en_US index data directory.
```
stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt
```
If you need to change the stopwords.txt file that is under workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common directory, the new stopwords.txt is copied to the Liberty_installdir\resources\search\index\managed-solr\config\v3-core-extension\common directory as part of setting up your WCB and CI/CD pipeline.
For example, in previous HCL Commerce verions you added a new English stop word: can. The following steps show you how to migrate this customization.
1. Add can to the bottom of your HCL Commerce Version 9 workspace_dir\search-config-ext\src\index\managed-solr\config\v3\common\stopwords.txt file.
2. Add the customized stopwords.txt path in the CONFIG column of SRCHCONFEXT table by running the following SQL statement:
```
Update srchconfext set config='stopwords_en=../../../../managed-solr/config/v3-core-extension/common/stopwords.txt' 
where indextype='CatalogEntry' and indexscope='10001';
```
Migrate your customized preprocess files.
In previous HCL Commerce versions, when you created a customized preprocess.xml file to index external data, the customized file is under the solrhome/pre-processConfig/MC_MasterCatalog/Dbtype directory. In HCL Commerce Version 9, such preprocess XML files reside under the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory.
Copy the wc-dataimport-preprocess-custom.xml file from your previous environment to the WCDE_installdir/xml/search/dataImport/v3/Dbtype/ directory on HCL Commerce Version 9. Also, copy any files that are referenced by the wc-dataimport-preprocess-custom.xml file to HCL Commerce Version 9, and update the wc-dataimport-preprocess-custom.xml file to ensure that it points to correct path file if you change the file location on HCL Commerce Version 9. For example, maybe you specified an input file in the wc-dataimport-preprocess-custom.xml as follows.
```

<_config:property name="inputFile" value="W:\WCDE_INT70\bin\Ratings.xml"/>
```
If so, you must copy the Ratings.xml file from previous environment to HCL Commerce Version 9.0.0.1+, and put it under location WCDE_installdir\bin, and update wc-dataimport-preprocess-custom.xml to point to the new location.
```

<_config:property name="inputFile" value="W:\WCDE_v9\bin\Ratings.xml"/>
```
Migrate your customized data import handler files.
If you wanted to index more data in previous HCL Commerce versions, you changed the wc-data-config.xml file in the specific core directory. For example, if you want to index more data from the X_RATING table in the English language, you edited the wc-data-config.xml file under the en_US core directory to allow query and deltaImportQuery to join the X_RATING table. In HCL Commerce Version 9, the Solr DataImportHandler (DIH) gets data from the view VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# and X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag#. Add this new customizable table into the view X_VI_CE_#INDEX_SCOPE_TAG#_#lang_tag# in wc-dataimport-preprocess-x-finalbuild.xml. For more information, see Configuring the Data Import Handler mapping.

After the customized tables are added into X_VI_CE table, customized data can be returned and made accessible to Solr by the DIH. Define the field-column mapping so that Solr can transfer the new customized column into Solr fields. Update the search-config-ext/src/main/resources/index/managed-solr/config/v3/CatalogEntry/x-data-config.xml file to add the field mapping.
The following is an example of the new mapping. With this mapping, the RATING column from the X_VI_CE view maps to the customerRanking field in the catalog entry index.
```
<field column="RATING" name="customerRanking"/>
```
Restart your Search server.
Build your Search index.