Automating the CI/CD pipeline

In the final stage in the CI/CD pipeline you package and deploy your customizations using a script automation tool. A number of tools are available to do this; in this example, we use Jenkins.

Building a Jenkinsfile to automate your processes

The Jenkinsfile sets the execution sequence of your build process. It includes the following stages and steps:
Note: You can skip a stage if you do not have customizations for that stage.
Stage 1: Package and deploy custom NiFi processors
This stage consists of the following steps:
  1. Extract the code from your filerepository.
  2. Build the source code of the Custom Java Processors.
  3. Package the code of the Custom Java Processor in a .nar file. Using your preferred build automation utility like Jenkins+Maven plugin create a script that calls the pom.xml targets that will build and package the code into a .NAR file.
  4. Create a new custom NiFi container with a copy of the NAR file.
  5. Create a custom image of the .nar file that needs to be copied into the NIFI docker container, using a Dockerfile, 
    COPY commerce-custom-search-bundles.nar /opt/nifi/nifi-current/extensions/commerce-custom-search-bundles.nar
  6. Push the custom image into the Docker Registry.
  7. Deploy and Run the new custom NiFi image to see the available processor from the NIFI UI.
Stage 2: Package and deploy custom NiFi process groups
This stage consists of the following steps:
  1. Extract the code from your file repository.
  2. Using a Dockerfile, build a custom image of the NIFI Registry container with the JSON files in it.
  3. Deploy the custom NiFi Registry image.
  4. Start the custom NiFi Registry image.
    Note: Creating a new NiFi Registry is not a requirement. You can also implement a custom separate NiFi Registry that holds the custom process group.
  5. In the same Dockerfile used in step 2, include import_flow.sh utility to import the process groups. Refer to Import the connectors into runtime registry
Stage 3: Package and deploy Ingest connectors
  1. Extract the code from your file repository.
  2. Using the following Curl command, upload the modified connector definition JSON file into the Ingest container to persist the descriptor into ZooKeeper.
    post /connector/<*CONNECTOR_NAME*>/upgrade/

    Example : Here is s Sample connection upgrade using Curl command:

    curl -X POST "http://v91tk.hclcdev.com:30800/connectors/someid.sample/upgrade" -H "accept: */*" -H "Content-Type: application/json" -d "{\"id\":\"string\",\"name\":\"string\",\"description\":\"string\",\"environment\":[\"string\"],\"created\":\"string\",\"modified\":\"string\",\"schedule\":{\"enabled\":true,\"startDate\":\"string\",\"endDate\":\"string\",\"startTime\":\"string\",\"everyInterval\":\"string\"},\"pipes\":[{\"name\":\"string\",\"label\":\"string\",\"version\":0,\"properties\":[{\"name\":\"string\",\"value\":\"string\",\"scope\":{\"name\":\"string\",\"sourceName\":\"string\",\"destinationName\":\"string\",\"type\":\"CONTROLLER_SERVICE\"}}]}],\"schemas\":[{\"name\":\"string\",\"schema\":{}}]}"
Customize properties and configuration of the default/existing components (update)
Make property and configuration changes to the process groups, processors, connections, and controller services. To make these changes, place the update descriptor JSON file into the specified volume mount folder (/profile/apps/search-ingest.ear/search-ingest.war/WEB-INF/classes/deployments/customization). Based on this, the Search Ingest Service API automatically runs the Update API with this update descriptor JSON provided for customizing the properties and configuration of the default Ingest connectors.