Create a custom NiFi processor by extending the default Ingest processor in the NiFi Toolkit

To set up the toolkit environment with the provided sample source code commerce-custom-search-marketplace-seller.zip in the same workspace as commerce-custom-search-bundles-9.1.x.x as explained in the pre-requisite follow the steps mentioned below. And for more information on commerce-custom-search-bundles-9.1.x.x see Tutorial: Customizing default Connectors with Ingest Profile.

1. Extract commerce-custom-search-marketplace-seller.zip to a working directory, commerce-custom-search-marketplace-seller, that contains a Maven project with a pom.xml file inside it.
2. Copy the following JAR files obtained from the NiFi container to the commerce-custom-search-marketplace-seller/libs project directory that was extracted from the project commerce-custom-search-marketplace-seller.zip. For more information, refer to steps 3 through 6 of Custom NiFi processors.

   Example from V9.1.10.0:

   1. commerce-search-processors-9.1.10.0.jar
   2. hcl-cache-core-9.1.10.0-20220322.160926-24.jar
   3. cf-base-9.1.10.0-20220322.160733-7.jar
   ingest-api.jar, which was included in the commerce-custom-search-marketplace-seller.zip project template, is already present in this folder.

   Note: From V9.1.12.0 onwards, ingest-api.jar is not made available inside tutorial assets.
3. Update pom.xml file in the root directory of the commerce-custom-search-marketplace-seller project and change the file name references to the JAR file names.
   Example from V9.1.10.0:

   <dependency>
       <groupId>com.hcl.commerce</groupId>
       <artifactId>dummy-commerce-search-processors</artifactId>
       <version>9.1.10.0</version>
       <scope>system</scope>
       <systemPath>${project.basedir}/libs/commerce-search-processors-9.1.10.0.jar</systemPath>
   </dependency>
   <dependency>
       <groupId>com.hcl.commerce</groupId>
       <artifactId>dummy-hcl-cache-core</artifactId>
       <version>9.1.10.0</version>
       <scope>system</scope>
       <systemPath> ${project.basedir}/libs/hcl-cache-core-9.1.10.0-20220322.160926-24.jar</systemPath>
   </dependency>
   <dependency>
       <groupId>com.ibm.commerce.cf</groupId>
       <artifactId>dummy-cf-base</artifactId>
       <version>9.1.10.0</version>
       <scope>system</scope>
       <systemPath> ${project.basedir}/libs/cf-base-9.1.10.0-20220322.160733-7.jar</systemPath>
   </dependency>
   

4. Import the extracted project as a Maven project into eclipse (File > Import > Maven > Existing Maven Projects) and select the root directory commerce-custom-search-marketplace-seller. Click Finish. Wait for the dependencies to complete downloading.

   

   This project has an example custom expression provider class ChangeSQL which implements the IngestExpressionProviderinterface. The ChangeSQL class extends the SQL to include a “DESCRIPTION” column from the SELLERDESC table. You can use this sample reference to learn how to modify the SQL used in the default Extract, Transform, and Load (ETL) stages.

   ComposeDatabaseSQLTest is the JUnit test that can be used for verifying the code logic of ChangeSQL.

   ChangeDocument is an Ingest profile post-processor extension that demonstrates how you can perform further data manipulation following the default transformation, in one of the Product related Ingest stages. CreateProductDocumentFromDatabaseTest is the JUnit test that can be used to verify the code logic inside of ChangeDocument. In this class, add an additional field “seller_description” with the value from the “DESCRIPTION” column of the MPSELLERDESC table into the document that is to be indexed in the Elasticsearch.

   In the Project explorer, Right-click project commerce-custom-search-marketplace-seller-- > Run As > Maven build... (Note that there are two menu items, Maven build and Maven build... select Maven build...) The Edit Configuration window opens.

   Enter the command clean install -U -Denforcer.skip=true into the Goals textbox and click the Run button.

   Once the build is finished, the jar file will be available in the target directory of this project, for example: commerce-custom-search-marketplace-seller-9.1.x.0.jar. For more information, see Custom NiFi processors.

   

In this topic, you have created a custom NiFi processor by extending the default Ingest processor using the NiFi Toolkit. In the next section, you will test the custom ingest logic in NiFi Toolkit before it can be deployed on the development or production environment.