Creating and indexing shards

This task sets up and indexes a specified number of sharding cores using the search index setup utility (setupSearchIndex), an input properties file, and the shard indexing utility (di-parallel-process).

Note: .

Before you begin

Interim fixes for Feature Pack 7 are required to use this feature. Apply JR50129, or the latest cumulative interim fix.
Decide on the sharding configuration you want to use. For example, either only horizontal shard types, or a combination of horizontal and vertical shard types.
Decide on the number of shards to create. An extra vertical shard is created by the setupSearchIndex utility when the configParallelShards action is used.

About this task

During this task, you perform the following high-level steps:

Set up the sharding cores by running the setupSearchIndex utility and specifying the shard-specific actions and parameters.
Create and populate the input properties file that will be used to process indexing shards.
Run the di-parallel-process utility to index the shards.

Procedure

Set up the sharding cores

Complete one of the following tasks:
- Log on as a WebSphere Commerce non-root user.
- Log on with a user profile that has *SECOFR authority.
- Log on with a user ID that is a member of the Windows Administration group.
Go to the following directory:
- WC_installdir/components/foundation/subcomponents/search/bin
- WCDE_installdir\components\foundation\subcomponents\search\bin
Run the search index setup utility:
- setupSearchIndex.bat -masterCatalogId masterCatalogId -indextype CatalogEntry -numOfShards numOfShards -action action -dbuser db_user -dbuserpwd db_password [-shardTags -shardTags]
- setupSearchIndex.bat -instance instance_name -masterCatalogId masterCatalogId -indextype CatalogEntry -numOfShards numOfShards -action action -dbuser db_user -dbuserpwd db_password [-shardTags '-shardTags']
- setupSearchIndex.sh -instance instance_name -masterCatalogId masterCatalogId -indextype CatalogEntry -numOfShards numOfShards -action action -dbuser db_user -dbuserpwd db_password [-shardTags -shardTags]
Where:
action

Prepares the utility for certain actions, such as configuring the shards.

configHorizontalShards

Configures indexing for new horizontal shards.

configHorizontalShardsUpdate

Configures indexing for existing horizontal shards.

configParallelShards

Configures indexing for new parallel shards, which creates horizontal shards, plus one vertical shard.

configParallelShardsUpdate

Configures indexing for existing parallel shards.

configShardsReset

Deletes all existing shards.

masterCatalogId

The ID of the master catalog (for example, 10101).
If you do not know the master catalog ID, run the following SQL for your starter store:
SQL: select * from catalog where IDENTIFIER='STORE_IDENTIFIER'
To find the master catalog ID for an Extended Site store:

Find the store ID:
select * from storeent where IDENTIFIER='STORE_IDENTIFIER'

Use the storeent_id as the store_id in the following SQL to find the Catalog Asset store ID of this Extended Site store:
select * from storerel where store_id=XXXXXX and streltyp_id=-4 and relatedstore_id not in (XXXXXX)
Where XXXXXX is the storeent_id.

Get the master catalog ID:
select * from storecat where storeent_id=YYYYYY and mastercatalog='1'
Where YYYYYY is the relatedstore_id.
numOfShards

The number of sharding cores you want to prepare.

dbuser

The name of the user that is connecting to the database.

dbuserpwd

The password for the user that is connecting to the database.

shardTags

Optional: Creates the specified shard tags.
For example, passing in -shardTags X,Y,Z creates the following shards: Shard-X, Shard-Y, and Shard-Z.
Note: The specified number of shard tags must match the specified number of shards. If not specified, a letter is used from A to Z. Therefore, numOfShards is up to a maximum of 26.
Verify that the preprocessing shards were created successfully according to the number of shards that are specified in the utility, under the following directory:
- WC_installdir/instances/instance_name/search/pre-processConfig/MC_masterCatalogId/db_type/Shards/Shard-#Shard-Tag#
Note: For a vertical configuration, the number of preprocessing shards that exists is the number of shards that are specified in the utility, plus one (numOfShards+1).
Verify that the indexing shards were created successfully according to the number of shards that are specified in the utility, under the following directory:
- WC_installdir/instances/instance_name/search/solr/home/MC_masterCatalogId/Shards/locale
Restart the WebSphere Commerce search server.
If you are using indexed contract prices:
1. Open the following file for all shards: WC_installdir/instances/instance_name/search/solr/home/MC_masterCatalogId/Shards/locale/CatalogEntry_#Shard-Tag#/conf/wc-data-config.xml.
2. Find all instances of the following table name TI_CNTRPRICE_0_#Shard-Tag#.
3. Replace them with the following table name: TI_CNTRPRICE_0.
  This update enables the utilities to look for contract prices in the TI_CNTRPRICE_0 table, instead of the shard-specific tables.
4. Save your changes.
5. Run the di-calculateprice utility. For specific usage, see Building the WebSphere Commerce search contract price index.

Prepare the input properties file

Create an input properties file to be used for indexing, based on the output of the setup utility.
The sharding input properties file is used by the di-parallel-process utility to process indexing shards. It contains the following sections of properties:
- System properties, which are shared by other utilities. For example, passwords that are common among utilities.
- Database properties, which are used to establish database connections with the database server.
- Global preprocessing and indexing properties, which are used for preprocessing and indexing by all shards.
- Master search server properties, which specifies the master index cores where all shard data is merged.
- Horizontal Shard properties, which specifies the horizontal shard properties.
- Vertical Shard properties, which specifies the vertical shard properties.
Use the following sample file for reference: di-parallel-process.zip
The sample uses the en_US locale with shards A, B, and C. It contains the following files:

di-parallel-process-FEP8-linux-oracle.properties

The sample sharding input properties file for a Linux operating system with Oracle.

password.properties

The sample password properties file, referenced by the sample sharding input properties file. It contains passwords encrypted by the wcs_encrypt utility.

For more information about the properties file and expected values, see Sharding input properties file.

Run the shard indexing utility

Go to the following directory:
- WC_installdir/bin
- WCDE_installdir\bin
Run the utility:
- di-parallel-process.bat input_properties_file
- di-parallel-process.bat input_properties_file -instance instance_name [-dbuser dbuser] [-dbuserpwd dbuserpwd] [-searchuser searchuser] [-searchuserpwd searchuserpwd]
- di-parallel-process.sh input_properties_file -instance instance_name [-dbuser dbuser] [-dbuserpwd dbuserpwd] [-searchuser searchuser] [-searchuserpwd searchuserpwd]
Where:

input_properties_file

The relative path of the input properties file to pass into the utility.

For example, ../../di-parallel-process-linux-oracle.properties

instance_name

The name of the WebSphere Commerce instance with which you are working (for example, demo).

dbuser

Optional: The name of the user that is connecting to the database.

dbuserpwd

Optional: The password for the user that is connecting to the database.

searchuser

Optional: The search application administrative user name.

searchuserpwd

Optional: The search application administrative user password.
Ensure that the utility runs successfully. Either check the exit code, or the wc-dataimport-parallel-processor.log file for more information.

What to do next

Replicate the merged index into the repeater, then to the other nodes.