HCL Commerce Version 9.1.2.0 or later

Collecting Elasticsearch based search-related MustGather data for Natural Language Processing (NLP)

Prepare the following MustGather information before you call HCL Support to help with the troubleshooting process.

Before you begin

This MustGather makes use of request-level tracing to only apply tracing for the Query REST request being used for the scenario, rather that applying service-level tracing which would apply tracing for all Query REST request being processed in the application. To enable request-level tracing, follow the steps in Logging and troubleshooting the Ingest and Query services.

About this task

This MustGather can be used to investigate the cause of Elasticsearch-based NLP issues. If you are trying to answer one of the following questions, this is the MustGather you will want to use:

  1. Why are my colors, dimensions, or price filters not being recognized in search term search?
  2. Why am I seeing only matches on a single term out of the full set of terms in my search phrase?
  3. Why is this search term being incorrectly classified (for example as an attribute, measurement, category, etc) when I have a different intention for this search term?
  4. How does NLP transform my search phrase, and what does this look like after all NLP analysis has completed?

Procedure

  1. Confirm the Query REST request that applies to the scenario you are investigating. If you aren't sure what that is, and you are using a React-based storefront, you can use your browser's Developer Tools network tab when navigating the storefront to reproduce the issue, and view the Query REST call or calls being made.
  2. Execute the Query REST request with an additional header "X-Log-Level=TRACE". Do not add this to your REST request through a URL parameter.
  3. Collect all the trace files generated in the Query service:
    /app/ESQueryService/logs/​ 
  4. Collect the JSON responses for the following REST requests:
    GET https://{{data_environment_hostname}}:30921/search/resources/api/v2/configuration?nodeName=colors&locale={{locale}} 
    GET https://{{data_environment_hostname}}:30921/search/resources/api/v2/configuration?nodeName=component&envType={{locale}} 
    GET https://{{data_environment_hostname}}:30921/search/resources/api/v2/configuration?nodeName=filter&locale={{locale}} 
    GET https://{{data_environment_hostname}}:30921/search/resources/api/v2/configuration?nodeName=uoms&locale={{locale}} 
    GET https://{{data_environment_hostname}}:30921/search/resources/api/v2/documents/profiles/?profileType=NLP 
  5. Collect the NER files from the Query service:
    /opt/WebSphere/Liberty/usr/servers/default/custom-ner-*.txt

What to do next

Review the initial Request
The start of the scenario is going to be processing of the incoming REST request to the Query service. The resource handler used will be based on the API invoked. For example, when performing a search on the storefront for "couch", this will make use of the V2ProductResource handler as a result of using an /api/v2/products API call. For example:
2023-06-13T20:17:29.024Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.search.rest.V2ProductResource.findProducts:157 - 
URL: https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005&currency=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice 
Search Profile Definition
The search profile contains the main contextual information you use for constructing the search query and then processing the search results. It is important to verify that the search profile matches the expected behavior for this search. To validate the search profile definition being used, you can look for the tracing output from SearchConfigurationRegistry.getSearchProfile(). For example, here is the search profile definition for HCL_V2_findProductsBySearchTermWithPrice:
2023-06-13T20:17:29.249Z [Default Executor-thread-52975] [7244267430895139116] 
TRACE c.h.c.s.i.c.SearchConfigurationRegistry.getSearchProfile:368 
- ENTRY profileName:HCL_V2_findProductsBySearchTermWithPrice resourceName:productview resourceURI:/api/v2/products?searchTerm

2023-06-13T20:17:29.249Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.c.SearchConfigurationRegistry.getSearchProfile:382 
- EXIT searchProfile: SearchProfile [parentProfileName=, profileName=HCL_V2_findProductsBySearchTermWithPrice, indexName=product, query=Query 
[params=[{DynamicKitReturnPrice=true}], queryFields=[default.search.text, default.search.normalized, default.search.text_*, default.sku.normalized, 
natural.keywords.normalized], sortFields={1=manufacturer.raw asc, 2=name.normalized asc, 3=offer.* asc, 4=offer.* desc}, responseFields=[id.catentry, 
id.store, buyable, identifier.sku.raw, name.raw, name.override.raw, description.raw, description.override.raw, url.thumbnail, url.override.thumbnail, 
keyword.text, keyword.override.text, manufacturer.raw, id.member, seller.raw, type, prices.*, path.*, relationship.item.id, relationship.product.id, 
identifier.mpn.raw, attribute.*, kit.preconfigured, kit.model, kit.URL, kit.default_configuration, kit.parent.model, kit.configurable, 
kit.parent.configurable, start, end, *_display, url.seo, url.override.seo], highlight=null, spellcheck={limit=5}]] 

If this does not match the expected profile definition, you can validate your configuration from the search profile Query API endpoint. For example, use the following endpoint to look up the HCL_V2_findProductsBySearchTermWithPrice configuration in the Auth environment:

GET https://data_environment_hostname:30921/search/resources/api/v2/documents/profiles/HCL_V2_findProductsBySearchTermWithPrice 
Natural Language Processing (NLP)
Commerce Search makes use of NLP to perform analysis on the search term or terms being used to identify intent behind the terms used and modify the search appropriately. If you are seeing unexpected results, it may be a result of this NLP analysis performing in a way that you may not expect, or it taking into account additional data points that you may not be considering for this search.

First, to identify the NLP providers that will be used for the search, you can look for the tracing output from SearchNLPSupportProvider.invoke(). For example, here is the tracing output when using the default NLP profile:

2023-06-13T20:17:29.752Z [Default Executor-thread-52975] [7244267430895139116] 
TRACE c.h.c.s.i.e.p.SearchNLPSupportProvider.invoke:107 
- Search NLP providers: com.hcl.commerce.search.internal.expression.provider.SearchNLPPartNumberProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPWhiteSpaceProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPCurrencySymbolProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPSpellCorrectionProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPExcludedTermProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPNumberFormatterProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPSTAExpansionProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPDependenciesParsingProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPMultiwordTermProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPLowerCaseProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPDMMProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPSpecialCharacterProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchMultiwordFilterProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPStopwordProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPWordToNumberProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPPriceFilterProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPPOSAndNERProviderHelper 
com.hcl.commerce.search.internal.expression.provider.SearchNLPColorMMProviderHelper 

If there is a particular NLP provider that you do not want to use, or you want to include your own NLP providers, see Natural Language Processor profiles for more information.

Next, you will want to review each of the NLP providers to see if it made the intended effect to the search. For example, if we have a synonym (Search Term Association or STA) between couch and sofa, we can confirm that SearchNLPSTAExpansionProviderHelper picked up this synonym by reviewing the tracing output for this NLP provider:

2023-06-13T20:17:30.158Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:164 - ENTRY
2023-06-13T20:17:30.159Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:165 - search term before STA expansion  : couch
...
2023-06-13T20:17:30.529Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.getNodeData:592 - nodeData : {sofa,couch=s}
2023-06-13T20:17:30.529Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.getNodeData:594 - EXIT
...
2023-06-13T20:17:31.794Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:468 - EXIT
2023-06-13T20:17:31.794Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.i.e.p.SearchNLPSTAExpansionProviderHelper.performSTAExpansion:470 - search term after STA expansion  : sofa couch 
ElasticSearch query
Since our ElasticSearch queries can be cached, we can use the tracing output from SearchResponseCache.myInvoke() to see the Elasticsearch query being used for our search. For example, here is the Elasticsearch query being generated for the earlier "couch" search:
2023-06-13T20:17:32.468Z [Default Executor-thread-52975] [7244267430895139116] 
TRACE c.h.c.s.i.e.p.SearchResponseCache.myInvoke:562 
- Cache name: com.hcl.commerce.search.internal.expression.processor.SearchExpressionProcessor Index name: 
auth.12001.product Source builder: {"from":0,"size":12,"query":{"function_score":{"query":{"bool":{"filter":[{"query_string":{"query":"workspace_name : \"Base\"","fields":[],"type":"best_fields","default_operator":"or",
"max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,
"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"-((type:product AND relationship.has_sku:true) OR (type:variant AND state:false)) AND (relationship.product.group:*)","fields":[],"type":"best_fields",
"default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,
"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"id.catalog:\"11501\"","fields":[],"type":"best_fields","default_operator":"or",
"max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,
"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"id.store:(\"11\" \"12001\")","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,
"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},
{"query_string":{"query":"id.language:\"-1\"","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,
"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"displayable:true","fields":[],"type":"best_fields",
"default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,
"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"(*:* AND -_exists_:facets.7000000000000001010.value.raw)","fields":[],"type":"best_fields","default_operator":"or",
"max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,
"boost":1.0}}],"should":[{"query_string":{"query":"(\"sofa\" OR \"couch\")","fields":["default.search.normalized^1.0","default.search.text^1.0","default.search.text_en_US^1.0","default.sku.normalized^1.0","natural.keywords.normalized^100.0",
"natural.nouns.normalized^100.0"],"type":"most_fields","default_operator":"and","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,
"phrase_slop":0,"lenient":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},{"query_string":{"query":"(\"sofa\" OR \"couch\")","fields":["default.search.normalized^1.0",
"default.search.text^1.0","default.search.text_en_US^1.0","default.sku.normalized^1.0","natural.keywords.normalized^100.0","natural.nouns.raw^100.0"],"type":"most_fields","default_operator":"and","max_determinized_states":10000,
"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"lenient":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],
"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}},"functions":[{"filter":{"match_all":{"boost":1.0}},"script_score":{"script":{"id":"boost-script-param-1","params":{"boostFactor":10.0,"param1":"couch"}}}}],
"score_mode":"sum","boost_mode":"sum","max_boost":3.4028235E38,"boost":1.0}},"explain":true,"_source":false,"stored_fields":["id.catentry","id.store","buyable","identifier.sku.raw","manufacturer.raw","id.member","seller.raw","type",
"prices.-11005.usd","path.11001","path.11501","path.11502","relationship.item.id","relationship.product.id","identifier.mpn.raw","attribute.*","kit.preconfigured","kit.model","kit.URL","kit.default_configuration","kit.parent.model",
"kit.configurable","kit.parent.configurable","start","end","*_display","prices.list.usd","prices.offer.usd","name.11.raw","description.11.raw","url.11.thumbnail","keyword.11.text","url.11.seo","url.12001.thumbnail","url.12001.seo",
"name.12001.raw","description.12001.raw","keyword.12001.text"],"track_total_hits":2147483647,"aggregations":{"itemCount":{"cardinality":{"field":"relationship.product.group"}}},
"suggest":{"correction":{"text":"couch","term":{"field":"default.correction.text","size":5,"suggest_mode":"MISSING","accuracy":0.3,"sort":"SCORE","string_distance":"INTERNAL","max_edits":2,"max_inspections":5,
"max_term_freq":0.01,"prefix_length":2,"min_word_length":4,"min_doc_freq":0.0}}},"collapse":{"field":"relationship.product.group","inner_hits":{"name":"data","ignore_unmapped":false,"from":0,"size":20000,"version":false,
"seq_no_primary_term":false,"explain":false,"track_scores":false,"stored_fields":["url.12001.thumbnail","url.11.thumbnail","id.*","prices.*"]}}} 

Since the entire Elasticsearch query is printed in the tracing, you could take this JSON query and execute it directly yourself against your product index. For example, here is how this same Elasticsearch query could be executed directly:

POST http://{{es_hostname_port}}/auth.12001.product/_search

{
    "from": 0,
    "size": 12,
    "query": {
        "function_score": {
            "query": {
                "bool": {
                    "filter": [
                        {
                            "query_string": {
                                "query": "workspace_name : \"Base\"",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "-((type:product AND relationship.has_sku:true) OR (type:variant AND state:false)) AND (relationship.product.group:*)",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "id.catalog:\"11501\"",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "id.store:(\"11\" \"12001\")",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "id.language:\"-1\"",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "displayable:true",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "(*:* AND -_exists_:facets.7000000000000001010.value.raw)",
                                "fields": [],
                                "type": "best_fields",
                                "default_operator": "or",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        }
                    ],
                    "should": [
                        {
                            "query_string": {
                                "query": "(\"sofa\" OR \"couch\")",
                                "fields": [
                                    "default.search.normalized^1.0",
                                    "default.search.text^1.0",
                                    "default.search.text_en_US^1.0",
                                    "default.sku.normalized^1.0",
                                    "natural.keywords.normalized^100.0",
                                    "natural.nouns.normalized^100.0"
                                ],
                                "type": "most_fields",
                                "default_operator": "and",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "lenient": true,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        },
                        {
                            "query_string": {
                                "query": "(\"sofa\" OR \"couch\")",
                                "fields": [
                                    "default.search.normalized^1.0",
                                    "default.search.text^1.0",
                                    "default.search.text_en_US^1.0",
                                    "default.sku.normalized^1.0",
                                    "natural.keywords.normalized^100.0",
                                    "natural.nouns.raw^100.0"
                                ],
                                "type": "most_fields",
                                "default_operator": "and",
                                "max_determinized_states": 10000,
                                "enable_position_increments": true,
                                "fuzziness": "AUTO",
                                "fuzzy_prefix_length": 0,
                                "fuzzy_max_expansions": 50,
                                "phrase_slop": 0,
                                "lenient": true,
                                "escape": false,
                                "auto_generate_synonyms_phrase_query": true,
                                "fuzzy_transpositions": true,
                                "boost": 1.0
                            }
                        }
                    ],
                    "adjust_pure_negative": true,
                    "minimum_should_match": "1",
                    "boost": 1.0
                }
            },
            "functions": [
                {
                    "filter": {
                        "match_all": {
                            "boost": 1.0
                        }
                    },
                    "script_score": {
                        "script": {
                            "id": "boost-script-param-1",
                            "params": {
                                "boostFactor": 10.0,
                                "param1": "couch"
                            }
                        }
                    }
                }
            ],
            "score_mode": "sum",
            "boost_mode": "sum",
            "max_boost": 3.4028235E38,
            "boost": 1.0
        }
    },
    "explain": true,
    "_source": false,
    "stored_fields": [
        "id.catentry",
        "id.store",
        "buyable",
        "identifier.sku.raw",
        "manufacturer.raw",
        "id.member",
        "seller.raw",
        "type",
        "prices.-11005.usd",
        "path.11001",
        "path.11501",
        "path.11502",
        "relationship.item.id",
        "relationship.product.id",
        "identifier.mpn.raw",
        "attribute.*",
        "kit.preconfigured",
        "kit.model",
        "kit.URL",
        "kit.default_configuration",
        "kit.parent.model",
        "kit.configurable",
        "kit.parent.configurable",
        "start",
        "end",
        "*_display",
        "prices.list.usd",
        "prices.offer.usd",
        "name.11.raw",
        "description.11.raw",
        "url.11.thumbnail",
        "keyword.11.text",
        "url.11.seo",
        "url.12001.thumbnail",
        "url.12001.seo",
        "name.12001.raw",
        "description.12001.raw",
        "keyword.12001.text"
    ],
    "track_total_hits": 2147483647,
    "aggregations": {
        "itemCount": {
            "cardinality": {
                "field": "relationship.product.group"
            }
        }
    },
    "suggest": {
        "correction": {
            "text": "couch",
            "term": {
                "field": "default.correction.text",
                "size": 5,
                "suggest_mode": "MISSING",
                "accuracy": 0.3,
                "sort": "SCORE",
                "string_distance": "INTERNAL",
                "max_edits": 2,
                "max_inspections": 5,
                "max_term_freq": 0.01,
                "prefix_length": 2,
                "min_word_length": 4,
                "min_doc_freq": 0.0
            }
        }
    },
    "collapse": {
        "field": "relationship.product.group",
        "inner_hits": {
            "name": "data",
            "ignore_unmapped": false,
            "from": 0,
            "size": 20000,
            "version": false,
            "seq_no_primary_term": false,
            "explain": false,
            "track_scores": false,
            "stored_fields": [
                "url.12001.thumbnail",
                "url.11.thumbnail",
                "id.*",
                "prices.*"
            ]
        }
    }
}
Final version of REST response
To find the end of the REST request processing, you can look at the tracing output for AbstractSearchResource.executeSearch(). For example, here is the final version of the REST response for the "couch" search (condensed for brevity):
2023-06-13T20:17:33.782Z [Default Executor-thread-52975] [7244267430895139116] TRACE c.h.c.s.rest.AbstractSearchResource.executeSearch:505 
- Result : <200 OK OK,{catalogEntryView=[{hasSingleSKU=false, buyable=true, resourceId=https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005&currency=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice, 
uniqueID=14033, thumbnailRaw=/EmeraldCAS/images/catalog/livingroom/furniture/chair4_b1_350.jpg, thumbnail=/hclstore/EmeraldCAS/images/catalog/livingroom/furniture/chair4_b1_350.jpg, 
sellerId=7000000000000003501, parentCatalogGroupID=/10501/10502, manufacturer=Stonehenge, shortDescription=Very cozy short design single sofa., catalogEntryTypeCode=ProductBean, 
groupingProperties={groupOfferPriceRange=[Ljava.lang.Object;@682c66e4, groupListPriceRange=[Ljava.lang.Object;@c25e4f60, groupCount=4, groupOwner=14033, groupMaxPriceValue=749.99, 
groupHero=14035, groupMinPriceValue=749.99}, name=Stonehenge UltraCozy Single Sofa, partNumber=LR-FNTR-0004, storeID=11, seo={href=/stonehenge-ultracozy-single-sofa-lr-fntr-0004}, 
price=[{usage=Display, description=L, currency=USD, value=800.0}, {usage=Offer, contractId=-11005, description=I, currency=USD, value=749.99}], attributes=[{identifier=Color, 
attribute.group= , attribute.natural=["Brown","Dark Grey","Seaweed","Denim"], usage=Defining, values=[{sequence=[2.0, 10.0, 10.0, 10.0], identifier=[brown, darkgrey, seaweed, denim], 
unitOfMeasure=[one, one, one, one], unitID=[C62, C62, C62, C62], image1=[/EmeraldCAS/images/catalog/swatches/sw_brown.png, /EmeraldCAS/images/catalog/swatches/sw_darkgrey.png, 
/EmeraldCAS/images/catalog/swatches/sw_seaweed.png, /EmeraldCAS/images/catalog/swatches/sw_denim.png], value=[Brown, Dark Grey, Seaweed, Denim], image1path=[/hclstore/EmeraldCAS/images/catalog/swatches/sw_brown.png, 
/hclstore/EmeraldCAS/images/catalog/swatches/sw_darkgrey.png, /hclstore/EmeraldCAS/images/catalog/swatches/sw_seaweed.png, /hclstore/EmeraldCAS/images/catalog/swatches/sw_denim.png], 
uniqueID=[7000000000000003002, 7000000000000003010, 7000000000000003014, 7000000000000003020]}], 
displayable=true, merchandisable=true, searchable=true, sequence=1.0, storeDisplay=false, name=Color, attribute.store=12001, facetable=true, comparable=true, uniqueID=7000000000000000501, 
swatchable=false}]}, ... ], displayable=true, merchandisable=true, searchable=true, sequence=1.0, storeDisplay=false, name=Color, attribute.store=12001, facetable=true, comparable=true, 
uniqueID=7000000000000000501, swatchable=false}]}], recordSetComplete=false, recordSetCount=12, recordSetStartNumber=0, recordSetTotal=22, 
resourceId=https://search:30901/search/resources/api/v2/products?storeId=11&searchTerm=couch&limit=12&offset=0&contractId=-11005&currency=USD&langId=-1&profileName=HCL_V2_findProductsBySearchTermWithPrice, 
recordSetTotalMatches=22, resourceName=productview, facetView=[], metaData={price=1}},[]>
NLP metrics in REST response
The metaData section of the REST response will capture various NLP changes that were made to the search, to help provide context about the results based on the original search phrase. For example, here is the metaData example from searching for "red couch under 600":
    "metaData": {
        "price": "1",
        "nlpParsing": [
            {
                "searchTerm": "red couch under 600",
                "sta": "[couch : [\"sofa\" OR \"couch\"]]",
                "price": "under 600 --> [LTE:600]",
                "pos": "ADJECTIVE --> [red]",
                "color": "COLOR --> [red]"
            }
        ]
    }