Enabling global tracing ID

You can use a global tracing ID as well as customized fields in your logging files. The global ID makes problem determination easy, and you can use it with all your servers, including Liberty and the WebSphere Application Server.

HCL Commerce Server uses a modernized micro-service architecture, which includes the Transaction server, Search server, Store server, Customization server, and Utility server. This design provides numerous benefits, including an independent development environment, a sophisticated build and deployment cycle, more reasonable access volume distribution, and clear separation for customization, among other things. This does, however, increase the complexity of problem determination. The servers are separated, and they need service invocation for interaction. A large volume of requests passes through the servers, and it can be difficult to get a comprehensive picture of which servers participate in resolving any given request. Finding the root cause of a problem in a distributed system can be difficult using a logging system that is designed for a monolithic architecture.

HCL Commerce Version 9.0.1.3 introduces an enhanced global tracing ID that improves problem determination. In addition to using the tracing ID, you can add customized fields into the logging file. You can set a maximum size (in megabytes) for each trace file, and set a maximum number of trace files in WebSphere Application Server and Liberty. These enhancements make it possible for you to integrate log files from different channels. By tracing the global tracing ID or customized fields, you can find the root cause of issues much more quickly.

Before you begin

Install Kafka Server and ZooKeeper server

Refer to the official guide of Kafka and ZooKeeper

Connect to Kafka and ZooKeeper

We can use the default run engine commands to configure the Kafka and ZooKeeper setting in transaction, search, and store server.

Configure Transaction Server to connect to Kafka and ZooKeeper

Run the command

set-kafka-server
                                            KafkaServers
                                        TopicPrefix
                                        listOfZooKeeper

, where

KafkaServers: List of Kafka servers, which are separated by a comma. Kafka IDs must follow the pattern hostname:port.
TopicPrefix: Kafka Topic Prefix.
listOfZooKeeper: List of ZooKeepers (zookeeperId1, zookeeperId2,.., zookeeperIdn) must be separated by commas. ZooKeeper IDs must follow the pattern: hostname:port

For example,

set-kafka-server 9.110.182.165:9092,9.110.182.166:9092,9.110.182.167:9092 
testTopic 9.110.182.165:2181,9.110.182.166:2181,9.110.182.167:2181

Refer to the following links for the parameter syntax used by the different server types:

Configure Store Server to connect to ZooKeeper

For example,

set-kafka-server testTopic 9.110.182.165:2181,9.110.182.166:2181,9.110.182.167:2181

Hint, the topic prefix must be the same as the one you create in the transaction server

Configure Search Server to connect to ZooKeeper

For example,

set-kafka-server testTopic 9.110.182.165:2181,9.110.182.166:2181,9.110.182.167:2181

Note: The topic prefix must be the same as the one you create in the transaction server.

About this task

There are two ways to configure the global tracing ID: by using the default REST API, or by updating the JVM configurations.

Procedure

Configure global tracing ID via the default REST API.
You can configure the global tracing ID in runtime environments by using the default REST API. For more information, see Extended Logging REST API Reference. The REST API can only be run by the SPI user.

To activate the ID, call the extended logging resource API to send the message to Kafka, and broadcast the message to all the servers. The method is PUT, and basic authentication for the spiuser ID and corresponding password must be included in the header.
```
PUT https://transactionServerHost:transactionServerPort/wcs/resources/extendedlogger/asyncconfig?enableFlag=true®istrationFlag=true&customFields=XEL_example
```
Where the REST API parameters are:

enableFlag

Used to enable or disable the extended logging. The value true will enable logging, while false will disable it.

registrationFlag

Used to register or unregister the custom fields. The value true means register, while false is used to unregister. When registrationFlag is false, and customFields is not used, all custom fields will be unregistered.

customFields

Used to register the custom fields added into extended log. This parameter must begin with “XEL_,". If you exclude this prefix, no custom information will be printed into the extended log. If you need to add many fields, separate them with commas. When a request is launched, add the customized fields into the header, and later, the customized fields will be printed in the extended log.

containerNames

Used to set the corresponding servers to consume the message. If this value is not set in the message, all the servers will response to the action. If it is set, only corresponding containers will respond and take effect. The value is the container host names, separated with commas.

Note: This method updates working memory for the server containers, but does not change the Docker images for the servers. Therefore, these configurations are not persistent; they will be lost after you restart the servers.
Make the global tracing ID persistent via JVM configurations.
You can turn on and configure Global Tracing using two JVM parameters:

global.tracing.enabled

Used to enable or disable extended logging. The value true is used to enable extended logging, while false disables it.

global.tracing.custom.fields

Used to register the custom fields. If many fields need to be added, separate them with commas.

By configuring your application docker logic, you can ensure that these parameters are automatically set during docker startup. This approach requires that you rebuild the server docker images to resolve the specific configuration requirement. For a tutorial on customizing the default docker image based on your specific requirements, Tutorial: Customizing the Docker start-up logic.
The run engine command set-system-property can be used to set the JVM parameters. Refer to the following links for the command details in different servers:

Results

When extended logging is enabled, a file is created that is named extended.log.0. When extended.log.0 reaches the maximum size, it is rolled to generate a new file extended.log.1. The number in file name is used to distinguish the rotated logs. The logs are created in the following directories:

/opt/WebSphere/AppServer/profiles/default/logs/**/extended.log.*
/opt/WebSphere/Liberty/usr/servers/default/logs/**/extended.log.*

Note:

The following text only applies to 9.1.10.0 and greater.

The extended login feature has been deprecated, and logs in JSON format are recommended instead.

Enable WebSphere Application Server logging with JSON output

If WebSphere Application Server traditional is configured for HPEL mode, log and trace data are output in binary format instead of plain text. Using HPEL LogViewer, log data can be continuously exported in JSON format and displayed in a Kibana dashboard.

WebSphere Application Server logs are output in plain text format by default, which is known as basic mode logging. For improved log and trace capabilities, upgrade the logging mode to HPEL (High Performance Extensible Logging).

The HPEL LogViewer is a WebSphere Application Server command-line tool for working with HPEL log data and trace data repositories. Filtering and formatting features in the tool make it easier to identify key material in data repositories.

In your WebSphere Application Server environment in VM Quickstarter, change the basic logging mode to HPEL mode.
Run the following LogViewer command, which outputs the log and trace data to the specified log file in JSON format and continuously monitors it.
```
<WAS_ROOT>/bin/logViewer.sh -outLog <logviewer-output-log-file> -format json -resumable -resume -monitor &
```

Enable WebSphere Application Server Liberty logging with JSON output

When the WebSphere Application Server Liberty messages.log file is output in JSON format, the output can be displayed in a Kibana dashboard.

A Liberty server's messageFormat logging property controls the output format of the messages.log file. By default, the messageFormat property is set to basic, resulting in plain text output. Changing the messageFormat setting to json enables the output to appear on the Kibana dashboard.

In the server.xml file of your Liberty server, if you want to messageFormat logging property in JSON then use logging messageFormat="json" and if you want your consolseFormat property in JSON then use consoleFormat="json" to indicate which sources to include in the output.

If you want to configure the CRS go to the following directory workspace_dir\Liberty\servers\crsServer\configDropins\overrides. Edit the custom configuration file server.xml. If the file does not exist, create it.

configDropins/overrides/server.xml
<logging messageFormat="json" consoleFormat="json" traceSpecification="com.ibm.commerce.component.*=all">
</logging>

The ExtendedLogging and GlobalTracingID feature can be enabled without Kafka by enabling the JSON logging provided by WebSphere Liberty and WebSphere Application Server. When you enable both of these logging options, you will get comparable traces as shown below;

In CRS application

{"type":"liberty_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_userDir":"W:\/IBM\/WebSphere\/Liberty\/usr\/","ibm_serverName":"crsServer","message":"Cannot find class impl with specified xpath: ComponentIntegrationServices\/Marketing\/@serviceImpl, will use default one instead.","ibm_threadId":"00000035","ibm_datetime":"2022-02-01T20:23:26.708+0530","module":"com.ibm.commerce.component.util.service.ComponentMarketingServiceFactory","loglevel":"WARNING","ibm_methodName":"initialize() executionContext:","ibm_className":"com.ibm.commerce.component.util.service.ComponentMarketingServiceFactory","ibm_sequence":"1643727206708_00000000000F2","ext_xRequestId":"ADHfu68Bvcs1kiA7","ext_thread":"Default Executor-thread-4"}

{"type":"liberty_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_userDir":"W:\/IBM\/WebSphere\/Liberty\/usr\/","ibm_serverName":"crsServer","message":"Configured name binding, com.ibm.commerce.foundation.server.services.search.contextpath, is not defined in server","ibm_threadId":"00000035","ibm_datetime":"2022-02-01T20:23:27.360+0530","module":"com.ibm.commerce.component.util.BackendServersConfigurationUtil","loglevel":"WARNING","ibm_methodName":"getComponentConfigThroughJNDI(String componentId, String configGroup, String propertyName, String defaultValue) executionContext:","ibm_className":"com.ibm.commerce.component.util.BackendServersConfigurationUtil","ibm_sequence":"1643727207360_00000000000F3","ext_xRequestId":"ADHfu68Bvcs1kiA7","ext_thread":"Default Executor-thread-4"}

In TS application

{"type":"was_message","host":"LP1-AP-51879416.PROD.HCLPNP.COM","ibm_cellName":"localhost","ibm_nodeName":"localhost","ibm_serverName":"server1","ibm_sequence":"1643727200393_0000000000C68","message":"CMN1501E: User -1002 does not have the authority to perform action \"AjaxAccountAddressForm\" on resource \"com.ibm.commerce.command.HttpForwardViewCommandImpl\" for command \"null\".","ibm_datetime":"2022-02-01T20:23:20.393+0530","ibm_methodName":"isAllowed","ibm_className":"AccManager","ibm_threadId":"000001bc","module":"com.ibm.websphere.commerce.log.CommerceSrvr","loglevel":"SEVERE","ext_appName":"WC","ext_xRequestId":"ADHfu68Bvcs1kiA7"}