Building the Elasticsearch index

Once you have installed the HCL Commerce core applications and set up an initial storefront environment, start the default Search Docker containers. Then, follow these instructions to connect the Version 9.1 Search microservices into a working system.

Procedure

  1. Build the Search index.
    The preferred approach is to make the index-building call directly to the Ingest server. For every index except the price index, you can get better details and more complete logs this way. First, determine the name of the connector you are building. You would usually build against auth.reindex, which is analogous to performing a complete index build with Solr. You can build against other connectors if you want to index specific subsets.
    HCL Commerce Version 9.1.12.0 or laterTip: If you are using the Marketplace feature, indexing by default waits for the postindex process to complete before updating the storefront, unlike in previous versions. There can be a delay before the storefront is refreshed if you have a large number of marketplace users. To prevent the storefront from waiting for postindexing to complete, set the flow.concurrent.postindex parameter to true instead of its default, false, using the following endpoint.
    http://Query_Server:30920/search/resources/api/v2/configuration?nodeName=ingest&envType=auth
    {
        "global": {
            "connector": [
                {
                    "name": "attribute",
                    "property": [
    {                         "name": "flow.concurrent.postindex", "value": "true"   }
                    ]
                }
            ]
        }
    }
    1. Build the index using the following REST call. This example assumes you are indexing the auth.reindex connector.
      POST https://ingestServerHostname:port/connectors/auth.reindex/run?storeId=11
      This call will return a runID. For example:
      {     "runId": "i-26144b4d-cd1c-4679-b2c2-870fc27e6095" } 
    2. You can check on the status of the build while it is ongoing. To do this, take the runID returned in Step 1.a and make the following REST call, inserting the runID after/runs/ in the string. Using the above runID, for example, the call would be:
      GET http://ingestServerHostname:port/connectors/auth.reindex/runs/i-26144b4d-cd1c-4679-b2c2-870fc27e6095/status
      The status information that is returned tells you approximately how much progress has been made in building the index.
      {     "date": "2022-04-06T17:51:44.462",
           "runId": "i-26144b4d-cd1c-4679-b2c2-870fc27e6095",
           "fromType": "Ingest",
           "message": "Indexing running, current progression of indexing is at process group: auth.reindex - WaitLink - Category Stage 1a",
           "status": -1,
           "progress": "30% (30 out of 101 pipes processed) " } 
      CAUTION:
      1. The percentage value provided in the progress field does not refer to time. In the above example, 30% does not necessarily mean that you are 30% of the indexing time has elapsed. Rather, this percentage is a measure of the number of pipes processed so far.
      2. With subsequent calls you may see the progress number rise or fall. For example, your first call may return a progress value of 60%, while a second one done a short time later may return 30%. This occurs because the indexing process is broken up into different catalogs, languages, and so on and , and does not process them all at once.
      When indexing is complete, the call produces a report summarizing the index, with any errors and special conditions noted. For example,
      "message": "Indexing run finished according to Nifi queue being empty for given connector. 
      {\"start\":\"2022-04-06T17:50:17.821Z\",\"end\":\"2022-04-06T17:53:02.346Z\",\"run\":\"i-26144b4d-cd1c-4679-b2c2-870fc27e6095\",\"severities\":
      {\"I\":15,\"W\":2}
      
      ,\"codes\":{\"DI1002I\":14,\"DI1050W\":2},\"locations\":{\"warning\":
      {\"Others\":2}
      
      ,\"info\":{\"Others\":14,\"Price Stage 2, Copy Contract Prices\":1}},\"elapsed\":{\"absolute\":{\"connector\":164525,\"summary\":{\"Attribute Pipeline\":\"4548\",
      \"Catalog Pipeline\":\"5155\",\"Category Pipeline\":\"13956\",\"Price Pipeline\":\"2911\",\"Product Pipeline\":\"21389\",\"Store Pipeline\":\"25336\",\"URL 
      By removing the /status at the end of the call, you can retrieve more detailed logs to review any errors. The price index is an exception; it can only be built against the Transaction server. This is because the price calculations are done on the Transaction server, so you still need to invoke the POST call against the Transaction server.
    Alternatively, you can use the following method to build the index.
    1. Issue the following command from within a REST client.
      POST: https://ingestServerHostname:ingestServerPort/connectors/auth.reindex/run?storeId=11
      Leave the Body empty, and use basic authentication with the username spiuser and the password spiuserPassword.
      Note:
      • The default password for the spiuser user is passw0rd for HCL Commerce 9.1.0.0 through 9.1.8.0, and QxV7uCk6RRiwvPVaa4wdD78jaHi2za8ssjneNMdu3vgqi for HCL Commerce 9.1.9.0 and greater.
      • It is essential to set your own spiuser password to secure your deployment. For more information, see Setting the spiuser password in your Docker images.

      In the example, the storeId is given as 11; replace this with the storeId for your own stores.

      The default values are:
      • AuroraESite: storeId=1
      • AuroraB2BSite: storeId=2
      • Emerald storeId=11
      • Sapphire storeId=12

      OR

      From a command prompt, issue the following command:

      curl --user spiuser:spi_plain_text_password --insecure -X POST "https://ingestServerHostname:ingestServerPort/connectors/auth.reindex/run?storeId=11"
      This call will return a runID value. For example:
      { "runId": "i-26144b4d-cd1c-4679-b2c2-870fc27e6095" }
    2. Wait for the index build to complete, or observe its status using its runID value.
      Issue the following GET REST command to obtain the build status.
      Use the returned value for runID from the previous step.
      GET http://ingestServerHostname:ingestServerPort/connectors/auth.reindex/runs/runId/status

      OR

      From a command prompt, issue the following command:
      curl --user spiuser:spi_plain_text_password --insecure -X GET "http://ingestServerHostname:ingestServerPort/connectors/auth.reindex/runs/runId/status"
      • If your build is in progress, a message similar to the following will provide details of its current status.
        {     "date": "2022-04-06T17:51:44.462",     "runId": "i-26144b4d-cd1c-4679-b2c2-870fc27e6095",     "fromType": "Ingest",     "message": "Indexing running, current progression of indexing is at process group: auth.reindex - WaitLink - Category Stage 1a",     "status": -1,     "progress": "30% (30 out of 101 pipes processed) " }
        Note: Progress percentage is a simple metric based on the number of pipes processed. It does not accurately reflect the progress in terms of time.
      • Once complete, a message similar to the following will detail the result of the completed build.
        "message": "Indexing run finished according to Nifi queue being empty for given connector. {\"start\":\"2022-04-06T17:50:17.821Z\",\"end\":\"2022-04-06T17:53:02.346Z\",\"run\":\"i-26144b4d-cd1c-4679-b2c2-870fc27e6095\",\"severities\":
        
        {\"I\":15,\"W\":2}
        ,\"codes\":{\"DI1002I\":14,\"DI1050W\":2},\"locations\":{\"warning\":
        
        {\"Others\":2}
        ,\"info\":{\"Others\":14,\"Price Stage 2, Copy Contract Prices\":1}},\"elapsed\":{\"absolute\":{\"connector\":164525,\"summary\":{\"Attribute Pipeline\":\"4548\",\"Catalog Pipeline\":\"5155\",\"Category Pipeline\":\"13956\",\"Price Pipeline\":\"2911\",\"Product Pipeline\":\"21389\",\"Store Pipeline\":\"25336\",\"URL 
        You can remove the /status at the end of the REST GET to obtain more detailed logs to review any encountered errors.
    3. Verify search index data.
      Using a REST client, issue the following POST commands to verify that the search index data are available. Ensure that the contents of the Body consist of { "query": { "bool": { "must": { "match_all": {} } } } }​.
      • Store data: http://ElasticSearchServerName:30200/auth.store/_search
      • Category data: http://ElasticSearchServerName:30200/auth.storeId.category/_search
      • Product data: http://ElasticSearchServerName:30200/auth.storeId.product/_search
  2. Verify that the Query service is running and can find data. Using a REST client, issue the following command:
    https://CommerceServerName​:30901/search/resources/store/storeId/categoryview/@top
    This call will return the top categories for the specified store.
  3. Build B2B Contract Price Index
    After you create Contracts with price adjustment, build the full index as in step #2, then build the contract price Index. Issue the following command from within a REST client.
    POST: https://CommerceServerName​​​:ts-app-port/wcs/resources/admin/index/dataImport/build?connectorId=price&indexSubType=Price&storeId=12
    HCL Commerce Version 9.1.2.0 or later
    POST: https://CommerceServerName​​​:ts-app-port/wcs/resources/admin/index/dataImport/build?connectorId=auth.price&indexSubType=Price&storeId=12
    Leave the Body empty, and use basic authentication: Username:spiuser and Password: spiuserPassword. In the example, the storeId is given as 12 (Sapphire); replace this with the storeId for your own stores.
    Check the build index status via the REST interface.
     GET: https://CommerceServerName​​​:ts-app-port/wcs/resources/admin/index/dataImport/status?jobStatusId=jobStatusId
    Where jobStatusId is the value of jobStatusId in the response from the above step. Use basic authentication with the username spiuser and the password spiuserPassword.
    Note:
    • The default password for the spiuser user is passw0rd for HCL Commerce 9.1.0.0 to 9.1.8.0, and QxV7uCk6RRiwvPVaa4wdD78jaHi2za8ssjneNMdu3vgqi for HCL Commerce 9.1.9.0 or greater.

    Following are Optional Parameters to make a more targeted price update.

    contractId
    Optional: Used to build prices for the specified contract IDs.
    Specify comma-separated contract IDs to index.
    If no contract IDs are passed in, a full recalculation is performed by default.
    Example: https://localhost/wcs/resources/admin/index/dataImport/build?indexSubType=Price&contractId=-10008,-10005
    catentry
    Optional: Used to build prices for the specified catalog entries.
    Specify comma-separated catalog entry IDs to index.
    If no catalog entry IDs are passed in, a full recalculation is performed by default.
    Example: https://localhost/wcs/resources/admin/index/dataImport/build?indexSubType=Price&catentryId=10006,10005
    currency
    Optional: Used to build prices for the specified currencies.
    Specify comma-separated currencies to index.
    Note: Ensure that the CURLIST table contains any additional currencies that you want built by the calculate price utility.
    If no currencies are passed in, a full recalculation is performed by default.
    Example: https://localhost/wcs/resources/admin/index/dataImport/build?indexSubType=Price&currency=USD,CAD
  4. Activate the Authoring and Production Environments with Push to Live (PTL) end to end. This step assumes that you have created the Push-to-Live connectors.
    1. Set envType​ to live in the input.properties configuration file. Deploy the HCL Commerce live environment by following the instructions in Deploying HCL Commerce Version 9.1 with Docker Compose (for non-production usage).
    2. To replicate search index to production environment, issue the following command from within a REST client.
      POST: https://CommerceServerName​​​:ts-app-port/wcs/resources/admin/index/dataImport/build?connectorId=push-to-live&storeId=1
      Leave the Body empty, and use basic authentication with the username spiuser and the password spiuserPassword.
      Note:
      • The default password for the spiuser user is passw0rd for HCL Commerce 9.1.0.0 to 9.1.8.0, and QxV7uCk6RRiwvPVaa4wdD78jaHi2za8ssjneNMdu3vgqi for HCL Commerce 9.1.9.0 or greater.
      • The storeId is given as 1 (AuroraESite). Replace this value with the storeId for your own stores.
      You can verify the status via the REST interface.
      GET: https://CommerceServerName​​​:ts-app-port/wcs/resources/admin/index/dataImport/status?jobStatusId=jobStatusId
      HCL Commerce Version 9.1.4.0Restriction: If Catalog data changes are not available in your live stores after a push-to-live operation, trigger a WCT+ESINDEX invalidation operation when you make the update. For more information about the caches to be updated and the procedure, see Troubleshooting: Index changes are not reflected in storefront after Elasticsearch push-to-live.
    3. For Management Center for HCL Commerce catalog changes, run a full index, run stagingprop as usual, then run PTL.