Tutorial: Customizing default connectors with Ingest Profile

This tutorial demonstrates how to use Ingest profiles with default connectors to completely separate custom configurations and logic code from the product release logic. This logical separation makes it easier to upgrade to future product releases.

In the default development environment, the search profile indexes the preset attribute seller for Marketplace stores. The seller is the vendor who sells the products on the marketplace store.

In this tutorial, you learn how to index an additional attribute seller description for the marketplace store by extending the default SQL in a custom Ingest processor. You will write custom Ingest processor code in Java and build and unit test this custom Ingest processor logic exclusively in a NiFi toolkit, to extend the default connectors. Overriding the default query search profile allows you to query on the new attribute , which is indexed using the custom Ingest logic in the response fields.

Skill level

This tutorial is intended for advanced HCL Commerce developers responsible for creating and updating customizations for HCL Commerce search. The steps are mainly based on Customizing Ingest profiles.

To complete this tutorial, you will need to be familiar with the following terms and concepts:

HCL Commerce search
HCL Commerce services
Ingest profiles
Ingest system
Java
Maven
NiFi
SQL

Time required

Expect this tutorial to take 3 hours to complete. The tutorial takes longer if you explore concepts that are related to this tutorial.

Prerequisites

Perform the following steps to prepare your system for the tutorial.

Ensure that you have an HCL Commerce development environment configured for the Elasticsearch data platform. For more information, see Setting up the HCL Commerce Developer Search environment.
Download and extract HCL_Commerce_Search_Bundle_9.1.x.x.zip from the HCL License and Delivery portal. After you unzip the file, locate the file commerce-custom-search-marketplace-seller.zip in the extracted directory \dependencies\nifi\Tutorials.
Extract commerce-custom-search-marketplace-seller.zip into a temporary directory in your development environment. This file is used to configure the NiFi toolkit. This file includes the sample Java code and Junit tests that you use in the tutorial.
Enable the Marketplace preview feature in the HCL Commerce Developer Tooling environment. For more information, see Enabling Marketplace in the HCL Commerce Developer Legacy environment.
Enable the Marketplace feature using the following Query service configuration endpoint.
```
PATCH http://QUERY_HOST:QUERY_PORT/search/resources/api/v2/configuration?nodeName=ingest&envType=auth 
```
with this request body:
```
{ 
"global": { 
"connector": [ 
{ 
"name": "attribute", 
"property": [ 
{ 
"name": "flow.marketplace", 
"value": "true" 
} 
] 
} 
] 
} 
}  
```
Note: All APIs mentioned throughout the tutorial must be executed by adding basic authentication for the spiuser user, where the user is spiuser, and the password is
```
QxV7uCk6RRiwvPVaa4wdD78jaHi2za8ssjneNMdu3vgqi
```
Ensure ESite Index Model is enabled by running the following SQL query:
```
SELECT * FROM STORECONF WHERE name = 'wc.search.CASIndexModel'
```
You should receive a false in the column VALUE for the storeconf table.
Trigger a full index for the EmeraldMP store (storeId=31) using the following Ingest endpoint.
```
POST http://INGEST_HOST:30800/connectors/auth.reindex/run?storeId=31 
```
Once the build is complete and the NIFI console shows 0/0 bytes queued data, execute the following GET commands using a REST client.
1. Check the response for the default search profile HCL_V2_findProductByPartNumber_Details, by executing the following GET call.
```
GET http://QUERY_HOST:30920/search/resources/api/v2/products?storeId=31&seller=7000000000000007002&partNumber=MPHBA031_310301 
```
2. Check the response for the default search profile HCL_findCatalogEntrySKUs by executing the following GET call.
```
GET http://QUERY_HOST:30920/search/resources/api/v2/products?storeId=31&seller=7000000000000007002&id=14503 
```
The response contains the seller attribute. At the end of the tutorial, you will be able to see an additional attribute, seller_description in the response for the above GET calls.
Prepare the development toolkit for NiFi following the instructions in Custom NiFi processors. The NiFi toolkit must be configured with Eclipse set to use the Java 1.8 compiler, with Maven Version 3.8.4 or above. Set the Java build path for the imported project to use JavaSE-1.8 libraries.
Patch the Nifi container on the data platform and obtain the latest JAR files from the running patched container before you configure and build tutorial assets in the NiFi toolkit. As an example, for the Version 9.1.10 environment the JAR files are:
- commerce-search-processors-9.1.10.0.jar
- hcl-cache-core-9.1.10.0-20220322.160926-24.jar
- cf-base-9.1.10.0-20220322.160733-7.jar
Initiate a Maven Build... on the root bundle project. In the Project Explorer, right-click on the root (commerce-search-custom-bundle-9.1.x.x) and select Run As > Maven build.... The Edit Configuration window opens.
Enter the command clean install -U -Denforcer.skip=true into the Goals entry field and click the Run button.
Once the build is successful, the JAR files can be viewed in the target subdirectory of the commerce-custom-search-processors project.