HCL Commerce Version 9.1.10.0 or later

Migrating Ingest service customizations

If you have reviewed the recent release changes made to the Ingest service that are described in Keeping your Ingest and Query services up to date, you can proceed to migrate your customizations. Use the following guide.

Before you begin

If you are upgrading to Version 9.1.10.0 or later from a previous version of HCL Commerce Search, upgrade your ZooKeeper configuration as follows:
  1. Verify that a /connectors node exists inside of Zookeeper container. If the node is empty then there is no need to do anything further.
  2. If the /connectors node is not empty, then delete all the default connectors from the ZooKeeper /connectors node, except those default connector descriptors that are customized.
  3. Do not delete the connector descriptors using the Ingest delete API. Doing that will delete the connector pipelines from NiFi as well. Instead, use the Zookeeper command line interpreter (CLI) to perform delete operations.
HCL Commerce Version 9.1.10.0 or laterNote: On every restart of the Ingest service, the system looks for connectors created on the NiFi console. If any of the connector pipelines are not present on the NiFi console, but the connector descriptor is available inside of Zookeeper, then the Ingest service will create the pipeline for that connectors descriptor in the NiFI console. This check is performed only for user-created connector descriptors. Ingest will ignore this operation for any customized connectors descriptor based on default connectors.

About this task

For all versions, perform the following steps before installing the latest NiFi and NiFi Registry images.

Procedure

  1. Backup NiFi
    If you have any custom connector configurations, you should first backup these settings before proceeding with the migration. Your existing connector descriptor may not be compatible with the latest default provided connectors. After the migration, download (GET using Ingest Service REST API) the latest connector configuration, manually compare and merge the connector JSONs and then re-apply the updated connectors.
    To backup the connector configuration before migration, use the Ingest Service API endpoint to fetch and download the Connector Descriptor JSON: 
    GET http://INGEST_HOSTNAME:INGEST_PORT/connectors
    Save this response with an appropriate name. For example, if you are using HCL Commerce Version 9.1.10, you could save the response of this API as connectors_v9110.json.
  2. Backup the NiFi registry
    Prior to migration, make a backup of all your existing flows inside of the NiFi Registry by exporting them. For example, go into your existing NiFi Registry container/pod and run the following command on each flow registered in the NiFi Registry:
    sh /opt/nifi-registry/scripts/export_flow.sh 
        FLOWNAME > /opt/nifi-registry/FLOWNAME_v9110.json 
  3. Deploy the latest NiFi and NiFi Registry images
    Docker-compose method with Docker
    1. Update [your docker-compose].yml file to use the latest NiFi/NiFi Registry images. There is no default external mount involved for NiFi and NiFi Registry, however, if you added a custom external mount, you should back up your data and will need to recreate it.
    2. Bash into your ZooKeeper container, run zkCli.sh to start the ZooKeeper command line interface, and perform deleteall /connectors to remove the existing connectors.
    3. Start your containers: docker-compose -f [your docker-compose].yml up -d
    Helm method with Kubernetes
    The following steps are needed to be performed before deploying your NiFi pod to enable your new container to use what is provided with the latest image (instead of re-using the existing NiFi metadata that is currently stored in your persistent volume):
    1. Bash into your existing NiFi container.
    2. Remove the /opt/nifi/nifi-current/dockerdata/.data.version file (or even everything in this dockerdata directory).
    3. Bash into your existing ZooKeeper container.
    4. Run zkCli.sh to start up the ZooKeeper command line interface, perform “deleteall /connectors” to remove the existing connectors configuration.
    Next, update your NiFi deployment configuration to point to the latest NiFi and NiFi Registry images. Wait for the pods to become healthy before proceeding to the next step.
Note: Perform the following steps AFTER deploying the latest NiFi and NiFi Registry images:
  1. Update NiFi
    After deploying the latest NiFi image, download (GET using Ingest Service REST API) the latest connector configuration, manually compare and merge the connector JSONs and then re-apply the updated connectors.
    1. If there are any NiFi processor customizations, copy the commerce-custom-search-processors-nar-9.1.x.x.nar to /opt/nifi/nifi-current/extensions directory inside of the NiFi container and restart the NiFi application.
    2. Use the same Ingest Service API endpoint to get the latest OOTB connector configuration: 
      GET http://INGEST_HOSTNAME:INGEST_PORT/connectors
      Save the response of this API, for example as connectors_v9110.json if you are using HCL Commerce Version 9.1.10.
    3. Compare both the JSONs to manually merge the changes and then apply the updates back to NiFi using a REST POST on Ingest API connectors endpoint providing the final merged JSON in the body. 
      POST http://INGEST_HOSTNAME:INGEST_PORT/connectors
  2. Update the NiFi registry
    Merge any applicable customization back into the NiFi Registry and load them back into the NiFi Console.
    1. Enter your new NiFi Registry container/pod.  Run the following command on each flow registered in the NiFi Registry:  
      sh /opt/nifi-registry/scripts/export_flow.sh FLOWNAME > /opt/nifi-registry/FLOWNAME_v9110.json 
    2. Compare JSONs between the releases to determine the properties/SQL statements that need to be modified and then perform the changes in the new version of the JSON (for example, FLOWNAME_v910_updated.json).
    3. Update your environment with these customizations that have been re-merged by importing them back into the NiFi Registry (where appropriate): 
      sh /opt/nifi-registry/scripts/import_flow.sh FLOWNAME /opt/nifi-registry/FLOWNAME_v9110_updated.json
    4. Use either of the following methods (I or II) to update the pipes/process groups in NiFi with a new version:
      Using Ingest Service Upgrade REST API
      Using the following URL, run Upgrade API from Ingest Service to get the latest version of the pipe/process groups for a particular connector: 
      http://INGEST_HOSTNAME:INGEST_PORT/connectors/{id}/upgrade
      1. Use an empty body.
      2. Run Upgrade API multiple times if the upgraded pipe is in more than one connector.
      Note: This method does not work with pipes in framework connectors, Bulk Services, Logging Service, etc.
      Using the NiFi Console
      1. In the NiFi console, navigate to the upgraded pipe/process group that is displayed with a red arrow icon.
      2. Right-click the pipe/process group and then navigate to Version > Change Version to choose the desired version from the list of versions.
      3. Click Change