Minimal Elasticsearch and NiFi Configuration

The minimum setup for Elasticsearch and NiFi is provided by default in your initial build. It uses the following configuration.

HEAP Sizes

NIFI Heap – 9GB 
Elasticsearch Heap   - 12GB

Processor groups threading changes

Nifi Flow > Logging Service

Nifi Flow>>Logging Service>>Generate Log Message: Configuration/Concurrent Tasks = 4 
Nifi Flow>>Logging Service>>Batch Bulk Document – Log: Configuration/Concurrent Tasks = 4

All other processors are set at Concurrent Tasks= 1

Nifi Flow > Bulk Service – Product

Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Post Bulk Elasticsearch-Product: Configuration/Concurrent Tasks = 16`
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Set stage.end.time : Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search> Post Refresh Elasticsearch: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Product>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4

Nifi Flow > Bulk Service – Price

Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Post Bulk Elasticsearch-Price: Configuration/Concurrent Tasks = 16
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Price>>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4

Nifi Flow > Bulk Service – Inventory

Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Post Bulk Elasticsearch-Inventory: Configuration/Concurrent Tasks = 16
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Set stage.end.time: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Route On Environment: Configuration/Concurrent Tasks = 4
Nifi Flow>>Bulk Service – Inventory >>Bulk Elastic Search>Retry Flow File: Configuration/Concurrent Tasks = 4

Nifi Flow > NLP Service

Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>Generate Query: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>>GET Elasticsearch: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>SCROLL Elasticsearch>> SCROLL Elasticsearch: Configuration/Concurrent Tasks = 6
Nifi Flow>>NLP Service>>Parse NLP Messages>>NLP Processor>>Transform Document-CoreNLP Ingest Processor: Configuration/Concurrent Tasks = 16

Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Track Bulk Request: Configuration/Concurrent Tasks = 4
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Post Bulk Elasticsearch: Configuration/Concurrent Tasks = 16
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Analyze Bulk Response: Configuration/Concurrent Tasks = 4
Nifi Flow>>NLP Service>>Parse NLP Messages>>Bulk Elasticsearch>> Retry Flow File: Configuration/Concurrent Tasks = 4

Bucket size changes

1i

Nifi Flow> live reindex- DatabaseProductStage1i: Variables:scroll.bucket.size=5000

1e
Nifi Flow> live reindex- DatabaseProductStage1e: Variables:scroll.bucket.size=5000

Refresh rate change

NiFi Flow> live reindex- StoreSchema >> Setup Elasticsearch Index Schema : Populate Store Index Schema

Note: Stop the processor, click and edit the json object, replacing the refresh_interval value with the desired value.