Persisting search index data to a Gluster file system

Learn how to use Kubernetes and GlusterFS to deploy HCL Commerce with a persistent volume for the search index.

Before you begin

Before you begin:
  • Ensure that you have HCL Commerce running on Kubernetes. This sample tutorial does not cover how to deploy HCL Commerce on Kubernetes. For more information, see Deploying HCL Commerce Version 9 on Kubernetes.
  • Ensure that you are using a CentOS or RedHat Linux distribution. The procedure in this sample tutorial is based on CentOS Linux. Modify any code as needed to fit your distribution.

About this task

By default, when you build the search index in your HCL Commerce Version 9 runtime environment, the indexed files are stored in the Search server Docker container. This poses an issue. If the search container is removed, for example when you upgrade your environment, then all the existing data in the container is lost. You will need to rebuild the search index to restore your store’s full functionality. To avoid having to rebuild the index, you can persist the indexed files to a network file system. There are many different ways you can set up a network file system, but this sample tutorial will use the following tools:

A) GlusterFS: An open source scalable network filesystem. It can create various types of volumes, such as distributed, replicated, distributed replicated, dispersed, and distributed dispersed.

B) Heketi: a provisioning tool that you can use to manage the lifecycle of GlusterFS volumes through RESTful APIs.

C) Kubernetes: provides two API resources to manage storage for persisting data: PersistentVolume (PV) and PersistentVolumeClaim (PVC).

– PersistentVolume is a piece of storage in the cluster, which has a lifecycle independent of any individual pod.

– PersistentVolumeClaim is a request for storage, which consumes PV resources.

Kubernetes supports two methods for provisioning a PV: Static and dynamic. In the static method, the PersistentVolume will be created in advance. In the dynamic method, the PersistentVolume is created based on the PVC’s requirement. In this sample, you will learn how to integrate Gluster storage for Kubernetes with Heteki to deploy HCL Commerce with a persistent volume to a network file system.


Set up a Gluster cluster and Heketi
  1. Prepare three nodes. This sample solution uses three nodes to create a Gluster cluster. If you do not want to use three, it is recommended that you have at least two nodes.
  2. Add the GlusterFS repository to your system.
    1. Create the following file.
    2. Add the following content to the CentOS-Gluster-4.0.repo file.
      # Please see for more information
      [centos-gluster40]name=CentOS-$releasever - Gluster 4.0
      [centos-gluster40-test]name=CentOS-$releasever - Gluster 4.0 Testing
  3. Install the Gluster server and client.
    yum install --nogpgcheck glusterfs glusterfs-server glusterfs-fuse  glusterfs-rdma nfs-ganesha-gluster
  4. Edit your Gluster server node basic configuration by adding a new disk to the Gluster server node. After the new disk is added, use the following command to check the new disk name. The new disk does not have the “Device Boot” record. In this tutorial, we take the new disk /dev/sdb as an example.
    fdisk -l.
    For example:
    Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk label type: dos
    Disk identifier: 0x000adb00
     Device Boot Start End Blocks Id System
    /dev/sda1 * 2048 1026047 512000 83 Linux
    /dev/sda2 1026048 201326591 100150272 8e Linux LVM
    /dev/sda3 201326592 209715199 4194304 82 Linux swap / Solaris
    Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
  5. Create and format a partition
    fdisk /dev/sdb
    mkfs.xfs /dev/sdb1  
  6. Enable the Gluster service.
    systemctl enable  --now glusterd
  7. Repeat steps 2-5 on the other nodes.
  8. Install the Heketi server and client on one Gluster server.
    yum install --nogpgcheck -y heketi heketi-client
  9. Generate a Heketi SSH key and use all default values.
    sudo –u heketi ssh-kegen
  10. Modify the /etc/heketi/heketi.json configuration file. Configure “ssh” as executor and add the keyfile and user in the ssh section. Here is a sample configuration.
    "_glusterfs_comment": "GlusterFS Configuration",
      "glusterfs": {
        "_executor_comment": [
          "Execute plugin. Possible choices: mock, ssh",
          "mock: This setting is used for testing and development.",
          "      It will not send commands to any node.",
          "ssh:  This setting will notify Heketi to ssh to the nodes.",
          "      It will need the values in sshexec to be configured.",
          "kubernetes: Communicate with GlusterFS containers over",
          "            Kubernetes exec api."
        "executor": "ssh",
        "_sshexec_comment": "SSH username and private key file information",
        "sshexec": {
          "keyfile": "/var/lib/heketi/.ssh/id_rsa",
          "user": "root",
          "port": "22",
          "fstab": "/etc/fstab"
  11. Edit the /usr/lib/system/system/heketi.service file. Change the start option to include 2 dashes at the beginning of “config” instead of only 1 dash. Change from -config=/etc/heketi/heketi.json to --config=/etc/heketi/heketi.json.
  12. Export the Heketi server address variable.
    export HEKETI_CLI_SERVER=http://localhost:8080
  13. Enable and start the Heketi service.
    systemctl enable -- now heketi
  14. Create a Gluster cluster on the Heketi server and get the cluster ID from the command output. This creates a cluster named wcs.
    heketi-cli cluster create wcs
  15. Add Gluster nodes to the cluster. You can get the node Id from command output.
        heketi-cli node add --cluster={clusterId} --zone=1 --management-host-name={gluster-node1-hostname} --storage-host-name={gluster-node1--hostname}
        heketi-cli node add --cluster={clusterId} --zone=1 --management-host-name={gluster-node2-hostname} --storage-host-name={gluster-node2--hostname}
        heketi-cli node add --cluster={clusterId} --zone=1 --management-host-name={gluster-node3-hostname} --storage-host-name={gluster-node3--hostname}
  16. Allocate devices for these nodes.
    heketi-cli device add --name=/dev/sdb1 --node={nodeID1}
    heketi-cli device add --name=/dev/sdb1 --node={nodeID2}
    heketi-cli device add --name=/dev/sdb1 --node={nodeID3}
  17. Install the Glusterfs client on all Kubenetes subordinate nodes.
    yum install --nogpgcheck glusterfs-fuse
A Glusterfs-Heketi environment is now ready. You can use the environment with Kubernetes to provision volumes dynamically.

Update Kubernetes environment

  1. Register the Gluster storage class on the Kubernetes system.
    1. Run the following command to create a YAML file.
      kubectl create –f gluster-storage.yaml
    2. Add the following content to the gluster-storage.yaml file. This content creates a Gluster storage.
         kind: StorageClass
         name: glusterfs
      annotations: "true"
         resturl: "http://{heketi-server-address}:8080"
         restuser: ""
         secretNamespace: ""
         secretName: ""
         volumetype: "none" //Choose the distributed volume type. 
                            //Note: 'replicate' type fails due to a Gluster issue. When Gluster replicates the file,
                            //      the ctime is modified, which will causes Solr validation to fail.
  2. Verify that the storage class was created successfully.
    kubectl get sc
  3. Create a PersistentVolumeClaim.
    1. Run the following command to create a YAML file.
      kubectl create –f search-master-pvc.yaml
    2. Add the following content to the search-master-pvc.yaml file.
      kind: PersistentVolumeClaim
      apiVersion: v1
           name: {tenantName}-{environment}-search-master-gluster-volume
           - ReadWriteMany
              storage: 2Gi
      storageClassName: glusterfs
  4. Verify that the PersistentVolumeClaim and PersistentVolume were created successfully.
    kubectl get pvc
    kubectl get pv
  5. Update your existing search-master helm chart to include a volume mount to the PersistentVolumeClaim. For example, volumes:
     - name: searchvol
                claimName: {tenantName}-{environment}-search-master-gluster-volume 
    Here is an example of a search-master helm chart with the volumes definition, based on HCL Commerce Version Docker containers.
    Note: If you are using HCL Commerce Version, you need to change – name: “CONFIGURE_MODE” to – name: “OVERRIDE_PRECONFIG”. Starting from HCL Commerce Version, the start up configuration parameters was changed from OVERRIDE_PRECONFIG to CONFIGURE_MODE.
    apiVersion: extensions/v1beta1
    kind: Deployment
     name: {{ .Values.Common.Tenant }}{{ .Values.Common.Environment_name}}{{.Values.Searchapp.Name}}
     namespace: {{ .Values.Common.Namespace }}
     replicas: {{ .Values.Searchapp.Replica }}
     component: {{ .Values.Common.Tenant }}{{.Values.Common.Environment_name}}{{.Values.Searchapp.Name}}
     - name: ts-app-check
     image: "{{ .Values.Common.Image_repo }}{{ .Values.InitC.Image }}:{{ .Values.InitC.Tag }}"
     imagePullPolicy: IfNotPresent
     - name: "COMPONENT"
     value: "ts-search"
     - name: "NAMESPACE"
     value: "{{ .Values.Common.Namespace }}"
     - name: "TENANT"
     value: "{{ .Values.Common.Tenant }}"
     - name: "ENVNAME"
     value: "{{ .Values.Common.Environment_name }}"
     - name: srach-app-master
     image: "{{ .Values.Common.Image_repo }}{{.Values.Searchapp.Image}}:{{ .Values.Searchapp.Tag }}"
     cpu: 0.5m
     memory: 2048Mi
     cpu: 2
     memory: 4096Mi
     - name: "VAULT_URL"
     value: "{{ .Values.Common.Vault_URL }}"
     - name: "WORKAREA"
     value: "/search"
     - name: "ENVIRONMENT"
     value: "{{ .Values.Common.Environment_name }}"
     - name: "VAULT_TOKEN"
     value: "{{ .Values.Common.Vault_token }}"
     - name: "TENANT"
     value: "{{ .Values.Common.Tenant }}"
     - name: "SOLR_MASTER"
     value: "true"
     - name: "SOLR_SLAVE"
     value: "false"
     - name: "ENVTYPE"
     value: "auth"
     - name: "LICENSE"
     value: "accept"
     - name: "VAULT_CA"
     value: "true"
     - name: "CONFIGURE_MODE"
     value: "Vault"
     - containerPort: 3737
     name: port3737
     - containerPort: 3738
     name: port3738
     protocol: TCP
     path: /search/admin/resources/health/status?type=container
     port: 3737
     - name: Authorization
     value: Basic {{ .Values.Common.SPIUser_PWD }}
     initialDelaySeconds: 5
     periodSeconds: 5
     port: 3737
     initialDelaySeconds: 600
     timeoutSeconds: 300
     - name: myvol
     mountPath: /search
     mounttype: glusterfs
     - name: searchvol
     claimName: {tenantName}-{environment}-search-master-gluster-volume 
  6. Redeploy HCL Commerce by installing the new helm chart. For example, helm install <parameters> <helm_repo_name>/<helm_package>.


You now have HCL Commerce Version 9 running on Kubernetes with a persistent volume. When you recreate containers, you will notice that the search index data is persisted to the GlusterFS volume and you do not have to rebuild the search index.
  1. Build the search index for your store. Perform these steps only when all containers are ready to use. For more information, see Deploying HCL Commerce Version 9 on Kubernetes.
  2. Visit the store to ensure that everything displays as expected.
  3. Delete the deployment.
    helm delete --purge <chartName>
  4. Redeploy the HCL Commerce environment (without creating a persistent volume claim).
    helm install <newChart>
  5. Visit the store again. Everything should display as expected without needing to rebuild the search index. If you do not see categories or products, then the search index was not persisted. Review the steps to ensure that you completed everything.