Customizing the Elasticsearch-based search solution

The topics in this section describe tasks performed by an application developer to customize the Elasticsearch-based search solution. You can extend Ingest Service and Query Service to meet your business requirements.

The Elasticsearch Search solution is one component of a unified data environment. This environment is designed as a generic data-management system. The environment includes several major components, including the gateway, data services, and the ZooKeeper configuration management system.

The Redis gateway and ZooKeeper/Registry support the data services. It is the data services that control Search behavior, so you customize Search by extending the Ingest data-uploading subsystem or the Elasticsearch-based Query subsystem. There are four main ways you can do this, as outlined below. In addition, you can develop migration and continuous-delivery strategies based on templates and examples provided by HCL.

Adding new Ingest connectors

You can add a new connector to the Ingest service. Doing this allows you to add a new NiFi pipeline and configuration to import and index new kinds of data. The new connector is stored in ZooKeeper and the registry.

New connectors can be created using your preferred text editor. You can usually modify an existing JSON template and pass the modifiied JSON object into the Ingest service API/connector to create the new connector.

See Creating a NiFi service connector.

Adding new Ingest processors

You can modify the Ingest flow, that is, the behavior of the pipeline that belongs to a connector.

The new processor is created by to adding new logic written in Java or another programming language. The new logic is made available as a deployable processor in the NiFi user interface, and is then included as part of the pipeline.

See Custom NiFi processors.

Modifying or adding Ingest data flows

You can interrupt the flow of an existing (default) pipeline, or create your own pipeline from within the NiFi subsystem.

See Custom NiFi processors.

Adding new Rest APIs to the Query service

The Query service manages search-related functions such as Natural Language Processing (NLP) and text relevancy. You can use the Query APIs to add your own services or extend existing ones.

See Extending the Query Service.

Migrating and deploying your extensions

Migration information is provided to help you bring customizations made for the Solr Search solution into the Elasticsearch system. The data services fully supports CI/CD practices and HCL provides a recommended strategy for developing your own Elasticsearch-specific CI/CD pipeline.

For migration instructions, see Migrating Elasticsearch customizations. For a guide to implementing an Elasticsearch CI/CD pipeline, see CI/CD strategy to deploy customizations to higher environment.