System alerts when licenses are not available

It is possible for the instance to get into a state where it is still running but there are insufficient licenses for the current CPU configuration. This may happen if there is a network issue that prevents contacting the license server. To understand how this situation can occur, and the remedial action required, the sequence of events for license checks is detailed below:

Every 10 seconds, the instance compares the acquired licenses with the number of CPU VPs. If the number of CPU VPs has not changed, then this license check does not require contacting the license server and a Trusted Storage area on disk, caches the licenses locally. If the number of CPU VPs has changed, or after 23 hours, when the license is within 1 hour of the expiration period, the instance will contact the license server to renew the license. If it cannot renew an expiring license, or acquire more licenses because of a change in the number of CPU VPs, then it will start issuing warnings. While being unable to reconcile the license count with the licence server, the number of held licenses will be set to zero.

The first warning will appear as a message in the online.log and will also trigger an alarm event, severity 3 (ATTENTION), class 29 (LICENSING), through the ALARMPROGRAM script. The instance will continue to try to obtain the necessary licenses every 10 seconds but it will not constantly emit warnings. The next warning will be 5 minutes after the license expired or became unavailable, then 10 minutes, then 20, and then 40 minutes. After these initial warnings, the alerts will continue to be produced on an hourly basis until 24 hours has passed from the time the licensing became non-compliant. If no licenses have been obtained during that time, then the server will write a message to the log to indicate it is shutting down, issue a severity 4 alarm (EMERGENCY), and then shutdown the instance.

In order to bring the server back online, it will be necessary to make licenses available again. If this requires changing the LICENSE_SERVER onconfig parameter, then the Trusted Storage cache may not be able to sync up with the FNO system and the instance will be unable to start. To recover from this state, it is necessary to remove the Trusted Storage area. This is a directory in $ONEDB_HOME/etc called .lic.<SNUM> - replacing <SNUM> with the SERVERNUM of the of the instance.

To aid troubleshooting of licensing issues, the environment variable IFX_FLEXLOG can be set (to any value between 1 and 4) before starting the instance. This will cause extra information regarding the licensing sub-system to be printed to the file pointed to by the instance’s CONSOLE parameter. For example, if there is a network issue preventing access to the license server then you would see the following messages:

2021-05-27 09:21:12 LICENSING: <ERROR> Send Binary Message
2021-05-27 09:21:12 LICENSING: <ERROR> Unable to get license information from server
2021-05-27 09:22:12 LICENSING: <ERROR> server http://hclsoftware.compliance.flexnetoperations.com/instances/5BYD9U9NF7P2/request: curl error – 28:
Connection timed out after 60001 milliseconds