Specialized pipes and Process Groups in NiFi
This topic provides description and purpose of the Specialized pipes and Process Groups in NiFi.
The Process Group is a technique for combining components into a logical construct in order to structure the DataFlow in a way that is easier to grasp from a higher level. Any complicated dataflow benefits from the use of process groups. Creating your own customized process group allows you to undertake data ingestion and transformation tasks that are tailored to your organizational requirements. The NiFi Specialized Pipes and Process Groups are as follows:
- Gateway Service
- All events from the Transaction server or internal NiFi events are sent through the gateway service. This pipe merges multiple events into a single flow file. From event data, this pipe sets environment type, index name, run id (ID of the specific pipeline run), time, and flow file attributes.
- Routing Service
- Based on the index name from Gateway Service this pipe transfers the flow file to corresponding connector. For example if the index name is auth.product, the flow file transfers to auth.product connector.
- Logging Service
- This pipe receives all logger input events from various pipes and makes a bulk call to the log index to write all of the logs.
- Run Service
- This pipe handles run logs and makes bulk POST calls to the run index in Elasticsearch. This information about the run index is required for run statistics.
- Terminal Service
- This pipe is called at the end of each connector. It calls log index to get details for that connector run and generate a summary report. This pipe also handles cache invalidation events from cache invalidation pipe.
- Bulk Service - {index}
- This pipe is called from different {index} related pipes to update or
create the store index by doing an Elasticsearch bulk POST request.
Note: Store is the index in this case. Catalog, Product, Attribute, Category, Description, Price, Inventory, and URL are all examples of indexes.
- Bulk Service - Dataload
- This is a bulk service that handles all the dataload related ingest operations against all the supported indexes.
- NLP Service
- This pipe preloads the NLP index as well as the NLP cache information. It also parses data from the product index into natural field information for the NLP index.
- Metrics Service
- This is an internal service used for collecting HCL Cache related performance metrics to be used by Prometheus.