Creating a custom index
You may want to incorporate searches for data that is specific to your business and is not related to the commerce catalog. Aternatively, you may want to extend the existing product index in some way. In such cases, you can create a custom index for Elasticsearch or for the Query service to use. You can use this custom index when appropriate to include extra data in your product or category search result.
You can construct a custom index in two ways. You can do it manually, using whatever development tools you are comfortable with; or, you can create it through the Ingest service. You can adapt an existing pipeline if you choose this second approach.
Manual construction
You can create your own custom index in any fashion. Your approach does not have to follow the HCL Commerce programming model. In this case you are just constructing your own query and sending it to Elasticsearch, then adding the results to the search result in the postprocessing Query stage. For example, consider a case where you have a user rating that you want to index using a separate custom index. Items in this index could be tied to a particular product. Include this rating in your response for a term search. Inside the Query service, in the postprocessing stage, you simply use Elasticsearch to look up the user rating in your separately constructed index.
Inside the Query service, you can build a custom extension to perform the query operation. This is defined in a custom profile. See Setting up your custom search profile for instructions.
There are multiple stages in a custom profile. The postprocessing stage happens after the Elasticsearch query is performed, but before you send the received response data back to the storefront. The postprocessor allows you to manipulate the response at this stage. Because of the paging feature, you can choose whether or not to inject the data into your current page. You do not have to perform this operation with every response.
Helper packages within each API exist to make customizing Elasticsearch calls easier. If you reuse an existing helper, you gain the advantages of its preprogrammed capabilities, such as automatically-enabled tracing for troubleshooting your queries.
Using the Ingest service
You can build a custom index by creating a connector in the Ingest service. Either extend an existing connector or build your own; see Creating a NiFi service connector for more information. Inside the new connector, add processors to manage the schema, the Extract, Transfer, and Load (ETL) phase, and the loading of the data into Elasticsearch.
For more information on building a custom pipeline, see Create a connector using Ingest.