Reverting use of DAOS tier 2 storage

HCL Domino does not currently provide an automated way to move DAOS objects from tier 2 storage to tier 1 storage. The following steps describe how to revert manually.

About this task

The overall steps are as follows. See the detailed steps for more information.

Disable DAOS tier 2.
Verify that there are objects in tier 2.
Copy all tier 2 objects to tier 1 storage.
Perform a quick resync.
Verify that DAOS no longer has tier 2 references.
Delete tier 2 objects.
Perform a full resync.

Procedure

Disable DAOS tier 2 storage:
1. On the DAOS tab of the Server document, disable the DAOS Tier 2 field.
2. Restart the Domino server on which you made the change:
```
restart server
```
3. When the server comes back up, enter the following command at the server console:
```
tell daosmgr status
```
  Verify that the output says DAOS Tier2 is Disabled.
  Note: Although tier 2 is disabled, existing objects that are stored there can still be downloaded by Domino, because its object index is still aware that those objects are stored in tier 2. No additional objects will be pushed to tier 2 at this point, though.

Verify that there are DAOS objects in tier 2 by issuing the following command at the server console: .

tell daosmgr objectinfo summary

If the output indicates that there are no tier 2 objects, skip the remaining steps in this procedure. The following example output indicates that there are 32 tier 2 objects:

[1998:0002-082C] DAOS TIERED STORAGE SUMMARY
[1998:0002-082C] TIER 1 STORAGE:
[1998:0002-082C] 	OBJECTS      - 0
[1998:0002-082C] 	MB           - 0 MB
[1998:0002-082C] TIER 2 STORAGE:
[1998:0002-082C] 	OBJECTS      - 32
[1998:0002-082C] 	MB           - 70 MB
[1998:0002-082C] TOTAL STORAGE :
[1998:0002-082C] 	OBJECTS      - 32
[1998:0002-082C] 	MB           - 70 MB
[1998:0002-082C] 	AVERAGE AGE  - 1521 days

Copy all tier 2 objects to tier 1 storage. These steps assume the use of AWS CLI (command line interface), which must be installed and configured. But you can copy using any tool that works with your S3 backend.
1. Determine the profile name. The AWS command line interface (CLI) typically get the S3 credentials from a file named "credentials" in the ".aws" subdirectory under your user home directory. The credentials file can have multiple credentials; each one has a profile name in brackets, such as [dominocos], an aws_access_key_id, and an aws_secret_access_key. If you are using the [default] credential, you needn't worry about the profile name. Otherwise you will need to specify the correct profile name in your S3 commands.
2. To gather additional required information, issue the following command at the server console:
```
 tell daosmgr status
```
  Gather the following information from the command output:
  - Base Tier 1 Storage Path: Shown as baseStoragePath in the output.
  - S3 Endpoint: Shown as DAOS Tier2 Endpoint in the output. If using AWS S3, the endpoint may not be required. If using MinIO, use the endpoint of your MinIO server, for example:https://minio.myserver.renovations.com.
  - S3 Bucket: Shown as DAOS Tier2 Bucket in the output.
  - S3 Server ID: Shown as DAOS Tier2 Server ID in the output.
3. (Optional) To list all of your tier 2 objects, compose a command of the following form. Be aware that the command may take an extended period of time to complete.
```
aws s3 [--profile <profile-name>] [--endpoint-url <s3-endpoint>] ls 
   s3://<s3-bucket>/obj/<s3-server-id>/<s3-server-id>/ --recursive
```
  Note: The case must match exactly for the source path, and on non-Windows systems, for the target path.
  
  For example:
```
aws s3 --profile dominocos ls 
   s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/ --recursive
```
  Example output that lists two objects. You are likely to have more.
```
2019-10-09 15:55:02    1777421 obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   040E4598A2A287490B8DD7784FEB7150B876313E001B1D64.nlo
2019-10-09 15:55:01    1771669 obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   0E597BF9C316422FCA869567755CD882C8DCC469001B06EC.nlo
```
4. Copy your tier 2 objects to tier 1. You can choose any DAOS subcontainer as the destination directory on your Domino server. In this example we assume 0001. Execute a command of the following form. The command may take an extensive period of time to complete.
```
aws s3 [--profile <profile-name>] [--endpoint-url <s3-endpoint>] cp s3://<s3-bucket>/obj/<s3-server-id>/
   <s3-server-id>/<base-tier1-storage-path>/0001 --recursive
```
  For example:
```
aws s3 --profile dominocos cp 
   s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/ C:\Domino\MyServer\Data\DAOS\0001 --recursive
```
  Example console output:
```
download: 
s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   1FAE990C593D66B44923C1C0E41740F9628DDD170000D902.nlo to .\
   1FAE990C593D66B44923C1C0E41740F9628DDD170000D902.nlo
download:
s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   11FCA4B81120D5650D67C9427C80D6B8256AFCF30000D902.nlo to .\
   11FCA4B81120D5650D67C9427C80D6B8256AFCF30000D902.nlo
```
Perform a quick resync by issuing the following command at the server console:
```
tell daosmgr resync quick force
```
This causes DAOS to refresh its index of objects in both tier 1 and tier 2 but does not do a full resync. If you issue a tell daosmgr status command, the results show catalogState = RESYNCING. DAOS continues to operate in this state, but doesn't prune any deleted objects.

verify that there are no longer tier 2 objects by issuing the following command at the server console:

tell daosmgr objectinfo summary

The output should indicate 0 tier 2 objects, for example:

[1D1C:0002-6170] DAOS TIERED STORAGE SUMMARY
[1D1C:0002-6170] TIER 1 STORAGE:
[1D1C:0002-6170]        OBJECTS      - 32
[1D1C:0002-6170]        MB           - 70 MB
[1D1C:0002-6170] TIER 2 STORAGE:
[1D1C:0002-6170]        OBJECTS      - 0
[1D1C:0002-6170]        MB           - 0 MB
[1D1C:0002-6170] TOTAL STORAGE :
[1D1C:0002-6170]        OBJECTS      - 32
[1D1C:0002-6170]        MB           - 70 MB
[1D1C:0002-6170]        AVERAGE AGE  - 1521 days

Delete tier 2 objects from S3 by issuing a command in the following form. This command is the same as the command in Step 3c to list tier 2 objects except you replace the ls flag with rm:

aws s3 [--profile <profile-name>] [--endpoint-url <s3-endpoint>] rm 
   s3://<s3-bucket>/obj/<s3-server-id>/<s3-server-id>/ --recursive

For example:

aws s3 --profile dominocos rm s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/ --recursive

Example console output:

delete:
s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   040E4598A2A287490B8DD7784FEB7150B876313E001B1D64.nlo
delete:
s3://aws-hcl-dominocos/obj/96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   96A6456FB0F12AE91178C9A8AE0DAB7DB6F1130D/
   11FCA4B81120D5650D67C9427C80D6B8256AFCF30000D902.nlo

Perform a full DAOS resync in either of the following ways.
- If there is a Domino directory Program document scheduled to run the tell daosmgr resync command, wait for it to run.
- Otherwise, issue the command tell daosmgr resync force at the server console. A full resync may take an extended period of time to complete, so running it during non-business hours is preferable.
Note: The tell daosmgr resync quick force command that you issued in Step 4 assured that DAOS has an index of all existing objects. But the command does not assure that all the reference counts on the objects are precise. Precise reference counts requires a full resync, which involves rebuilding the object index from scratch by iterating over all attachment references in all DAOS-enabled databases.