Resizing NSF persistent volumes

Resize storage space used on a persistent volume used with Component Pack for IBM Connections™.

Before you begin

Resizing the NFS persistent volumes requires you to delete the existing configuration and install it with new settings. Complete the following requirements before you start the process:

  • Ensure there is sufficient disk space on the server hosting the persistent storage to accommodate the additional space.
  • Schedule at least 1 hour of down-time to allow you to complete the manual update required for adding storage space.

About this task

When you deployed persistent volumes for Component Pack, you specified the capacity of each unit and the amount of storage that each service claims. For example, by default Elasticsearch configures 3 PVs with 10 GB of storage on each, and each pod makes a claim for 10 GB of the space.

During the lifecycle of your deployment, you might need to resize the volumes to provide additional space. For example, your Elasticsearch deployment might consume more storage space for logs and metrics data than you expected. For some types of persistent volumes, you can dynamically expand the storage as described on the Kubernetes documentation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims.

You cannot resize NFS volumes dynamically because this task is a manual process and requires you to schedule down-time to complete it. This task typically takes about 1 hour.

The following instructions describe how to resize the NFS persistent volumes associated with Elasticsearch from the default capacity of 10GB to 20GB. The process for other components is similar. Resizing persistent volumes involves the following tasks:
  1. Back up the stored data.
  2. Validate that the Reclaim Policy on the persistent volumes is set to Retain (update it if needed).
  3. Perform a helm delete operation on Component Pack applications that are associated with the volumes that you will resize.
  4. Delete the volumes.
  5. Install the volumes, and the persistent volume claims, with the additional storage.
  6. Install the applications associated with the persistent volumes.

Procedure

  1. Back up the stored data.

    This step ensures that you don't lose your stored data when you delete the persistent volumes in a later step. Run the following commands to copy data to a back-up directory; in this example, the data is stored in the pv-connections directory so the back-up directory is called pv-connections-backup:

    
mkdir -p /pv-connections-backup
cd /pv-connections
cp -avr . /pv-connections-backup
cd /
zip -r /pv-connections-backup/pv-connections-backup.zip pv-connections-backup/
  2. Validate that the Reclaim policy on the persistent volumes is set to Retain (update it if needed).

    When you install persistent volumes, the Reclaim policy defaults to Retain. Verify the setting and if necessary, change it by completing the following steps.

    Note: For information on Reclaim policies, see the Kubernetes documentation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming
    1. Check the current policy by running the following command: 
      kubectl get pv

      In the output, the RECLAIM POLICY field should display Retain for all persistent volumes. If so, skip to step 3. If any other policy is listed, continue to substep b. to update it.

    2. Locate the full list of parameters that you used when you installed the persistent volumes with the connections-volumes helm chart, so you can use the same settings.
    3. Run the helm upgrade command, using the settings that you just retrieved but setting the Reclaim Policy to Retain:

      Note that nfs.server is a required parameter.

      
helm upgrade \ 
connections-volumes extractedFolder/microservices_connections/hybridcloud/helmbuilds/connections-persistent-storage-nfs-0.1.0.tgz \
--set Parameters_used_during_original_Helm_Install,\
persistentVolumeReclaimPolicy=Retain

      For example:

      
helm upgrade \ 
connections-volumes helmbuilds/connections-persistent-storage-nfs-0.1.0.tgz \ 
--set nfs.server=10.3.4.2,\
persistentVolumeReclaimPolicy=Retain
    4. Verify the update by checking the Reclaim Policy again: 
      kubectl get pv

      Make sure that the RECLAIM POLICY field displays Retain for all persistent volumes. If this is not the case, set the policy again.

      Important: Do not proceed until the policy is set to Retain; otherwise you will lose data during the remaining steps.
  3. Perform a helm delete operation to remove the applications that are associated with the volumes that you will resize.

    This step removes the persistent volumes from service; expect approximately 1 hour of down-time.

    As part of the resizing procedure, you must delete the persistent volumes and persistent volume claims (in this example, the ones that are associated with Elasticsearch), and then recreate them with additional storage. The deletion of the persistent volumes and persistent volume claims will not terminate successfully if there are running pods that are using them, so you must delete the applications by completing the following steps.

    1. Run the following command to delete the Elasticsearch application from the persistent volumes: 
      helm del --purge elasticsearch
    2. Run the following command to verify that all es- (Elasticsearch) pods are not running: 
      kubectl get pods -n connections | grep es-
      In this example, connections is the name of the namespace. You can replace es- with the name of the application you want to delete:
      • For Zookeeper or Solr volumes, use orientme
      • For MongoDB volumes, use infrastructure
      • For Customizer, use mw-proxy

      When the pods are successfully stopped, there are no results to this command.

  4. Delete the volumes that you will resize by running a helm upgrade with the connections-volumes helm chart.
    1. Run the following command perform a helm upgrade operation and disable Elasticsearch:

      Note that nfs.server is a required parameter.

      helm upgrade \ 
connections-volumes extractedFolder/microservices_connections/hybridcloud/helmbuilds/connections-persistent-storage-nfs-0.1.0.tgz \
--set Parameters_used_during_original_Helm_Install,\
es.enabled=false

      In this example, all Components are installed and you will only delete Elasticsearch. Because you are not changing any other settings, use the same set of parameters that you used when you installed Component Pack. For example:

      
helm upgrade \ 
connections-volumes helmbuilds/connections-persistent-storage-nfs-0.1.0.tgz \ 
--set nfs.server=10.3.4.2,\
es.enabled=false
    2. Run the following command to verify that the persistent volumes and persistent volume claims associated with Elasticsearch (es-) no longer exist: 
      kubectl get pv,pvc –n connections

      The persistent volumes and persistent volume claims no longer exist, but the data will be Retained on the file system.

  5. Run another helm upgrade to install the persistent volumes and the persistent volume claims with the additional storage.
    1. Run the following command to perform the helm upgrade:

      Again, use the original set of parameters; this time you can enable Elasticsearch and specify new storage settings for it. Note that nfs.server is a required parameter. 

      helm upgrade \ 
connections-volumes extractedFolder/microservices_connections/hybridcloud/helmbuilds/connections-persistent-storage-nfs-0.1.0.tgz \
--set Parameters_used_during_original_Helm_Install,\
es.enabled=true,\
es.capacity.storage=20Gi,\
es.resources.requests.storage=20Gi
    2. Run the following command to verify that the persistent volumes associated with Elasticsearch now have a capacity of 20Gi: 
      kubectl get pv
    3. Run the following command to verify that the persistent volume claimes associated with Elasticsearch now have a capacity of 20Gi and are Bound: 
      kubectl get pvc –n connections
  6. Install the Component Pack applications that are associated with the persistent volumes that you just resized.
    Following the instructions in the appropriate topic for the Component you are working with: