Verifying index integrity

You can use a specialized Ingest check pipeline to verify that the Elasticsearch index contains all the required data. This pipeline verifies that all incoming data is correctly inserted in the index. The pipeline compares the results of the Ingest process with the original input to ensure reliable data intake.

The integrity check consists of two parts. In HCL Commerce Search prior to Version 9.1.12, the check uses the pipelines auth.validate and


                live.validate

, one for each environment. From Version 9.1.12 onwards, a second set of connectors, auth.validate.cas and


                live.validate.cas

, is available for users of the Catalog Asset Store (CAS) indexing model. For more information about CAS indexing, see Choosing your index model. Each pipeline performs the following checks:

Store index validation: This stage of the pipeline verifies the existence of all documents. It counts the supported languages, catalogs and currencies in each document, and checks default field values against the database. This check is based on the Store ID and language ID.
Product Index Validation: This stage checks document existence by counting the number of documents for each product, item, bundle, variant and dynamic document type. It counts the number of attributes and checks the SEO URL for each catengry ID. If a URL is not present in the product index, it searches for it in the URL index and then the database. In addition, this stage checks whether products have items linked to them and verifies that the category, path and path_name section exist in the index for each Catentry.
Category Index Validation: Documents are checked for existence based on store, language, and catalog. This stage counts the number of parent single categories linked to, as well as the number of facets, children for non-leaf objects. It also verifies that a path section exists in each document.
Attribute Index Validation: This stage verifies the existence of all documents and counts the number of comparable, facet.zero, facet.search, merchandisable, searchable, ribbon, and swatchwable attributes. It also counts attribute values for each attribute. These checks are based on the store and language.

API endpoints

To trigger integrity check, below ingest API is used:

```
https://ingest-host:ingest-port/connectors/auth.validate/run?storeId=storeIdt
```
This API generates a runId which is used in the following APIs.

To verify the run status of the previous call, use:

http://ingest-host:ingest-port/connectors/auth.validate/runs/runId/status

To check if there is any error in the run, use the following API call:
```
http://ingest-host:ingest-port/connectors/auth.validate/runs/<runId>
```

To check validation logs use the following API call:

http://ingest-host:ingest-port/connectors/auth.validate/runs/runId?logSeverity=V

Note: The logSeverity parameter is case sensitive. To extract the logs for the validation pipeline append a capital V.