RHSM troubleshooting checklist

Troubleshooting RHSM errors

Table 1. Overview of the RHSM troubleshooting checklist
What to check Errors or warnings encountered Possible causes and remediation steps
GPG key is imported. If the GPG is not imported in an endpoint, you might find this entry in the EDR log: warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Public key for httpd-devel-2.2.15-56.el6_8.3.x86_64.rpm is not installed. The GPG is not imported in an endpoint. Check the EDR log of the endpoint. The log displays if the GPG is not imported. If it is not, import the GPG through the command line or by using Fixlets specific to either the Patches for RHEL 6 Native tools site or the Patches for RHEL site.
The prefetch plug-ins timeout settings is set too low. You might encounter a failed Fixlet deployment and with "fail" indicated at the 
execute prefetch plug-in
line.		 Use a task to set the timeout to 30 minutes.
Ensure that your certificates can access Red Hat repositories. Error: Certs cannot access any Repos or the certificates are only able to access some of the required repositories. Run a quick repository access check.
Ensure that the entitlement certificates are placed in the correct folders. Patch deployment fails. The certificates might not be placed in the correct folders and sub-folders. Unnecessary metadata files must be removed. For more information, see the guidelines in the following section: Entitlement certificates and system identity certificate are placed in the correct folders.
Ensure that the entitlement certificates have the correct format. Patch deployment fails. The user might have entitlements that have the old formats. To check the certificate format, see the steps detailed in the following section: The version of RHSM entitlement certificates have the correct format.
Entitlement certificates are active and have not expired. Patch deployment fails one day after creating an entitlement certificate. Follow the methods in the section to verify that the certificates have not expired. If the subscription is expired, you must generate or attach a new subscription (entitlement) to the entitlement certificate. Regenerate the identity certificate if it is expired.
Entitlement certificates have the correct subscriptions (entitlements) attached. Certificates cannot access the required Red Hat base repositories. Attach the correct subscription (entitlement) to the Entitlement Certificate to get access to the required repositories. Follow the methods in the section to verify the subscription entitlements that are attached to your entitlement certificates.
Entitlement certificate can access the Red Hat base repositories. Certificates cannot access the Red Hat base repositories. Possible causes:
  • The certificates have expired.
  • The required subscription were not properly attached when the system was registered through Red Hat Subscription Management portal.
  • The network or proxy is blocking RHSMPlugin.exe from accessing the repositories.
Run an access test which is described in the section.
Error messages in RHSMPlugin.log You might find the following entry in the log:

ERROR : All Key and Cert pairs in 'rootCertDir' cannot access: https://cdn.redhat.com/content/dist/rhel/client/7/

7Client/x86_64/os/repodata/repomd.xml
  • If you don’t need to deploy patches to any such endpoints, you may safely ignore this message. This might be caused by a no hash limitation.
  • If the error message is associated with a repository that you need for your patch deployment, this error might be due to several reasons. Refer to the following section: Entitlement certificates are not able to access the repositories

The GPG key is imported

A GPG key must be imported from Red Hat to download Red Hat content. After deploying a patch, check the EDR log of the endpoint, which is located at var\opt\BESClient\EDRDeployData\EDR_DeploymentResults.txt.

If the GPG key for an endpoint is not imported, you might see the following warning in EDR_DeploymentResults.txt. 

warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID fd431d51:
NOKEY Public key for httpd-devel-2.2.15-56.el6_8.3.x86_64.rpm is not installed

To import the GPG key for the endpoint, use the following command: rpm --import /mnt/cdrom/RPM-GPG-KEY-r​edhat-release.

You may also use the following Fixlets to import the GPG key for the endpoint:
  • Patches for RHEL 6 Native Tools: 301 Import RPM-GPG-KEY-redhat-release - RHEL 6
  • Patches for RHEL 7: 301 Import RPM-GPG-KEY-redhat-release - RHEL 7
These steps usually only needs to be done once on each newly-set up endpoint.

Ensure that the timeout setting is sufficient to execute the prefetch plug-in

You might need to configure the plug-ins timeout setting if the Fixlet deployment fails and from the Action Script Execution Detail in the console, "fail" is indicated in the execute prefetch plug-in line.

From the Patching Support site, use this task to set the timeout to 30 minutes: Change Timeout for Prefetch Plugins.

After applying the task, restart the BES client with the following task from the BES Support site: TROUBLESHOOTING: Restart BES Client on RHEL/SUSE.

Your certificates can access Red Hat repositories

You can run a quick test to check if your certificates can access Red Hat repositories. Typically, the test runs less than 10 seconds.

The RHSM plug-in is usually located in C:\Program Files (x86)\BigFix Enterprise\BES Server\DownloadPlugins\RHSMProtocol.

Run the following repository access test. 
RHSMPlugin.exe --check-baserepos

There are three possible outcomes when running the repository access test. Possible outcomes from Repo Access Test:

None of the certs can be accessed
INFO     :  Base Repos Test Summary
INFO     :  Certs in <rootCertDir> can access 0 / 12 Base Repos:

ERROR    :  Error: Certs cannot access any Repos.
The certificates were not set up properly. Continue with the checklist. For more information about setting up and downloading both certificates, see Setting Up RHSM Certificates.
Certificates are able to access all required repositories.
INFO     :  Base Repos Test Summary
INFO     :  Certs in <rootCertDir> can access 3 / 12 Base Repos:

INFO     :  server-7-x86_64:      Red Hat Enterprise Linux 7 Server (RPMs)
INFO     :  server-6-x86:         Red Hat Enterprise Linux 6 Server (RPMs)
INFO     :  server-6-x86_64:      Red Hat Enterprise Linux 6 Server (RPMs)
Certificates are able to access only some required repositories.

You might need to patch endpoint types that do not appear in the list of successfully accessed repositories. For example, you have Workstaton endpoints but the output only shows access to the Server repositories. In such cases, you must attach the required subscriptions to the certificates through the Red Hat portal.

The list of repositories that are tested are derived from the RepoList (“primaryRepoListFile", “extendedRepoListFile") file that is referenced in the plugin.ini. At the time of writing, the list of repositories in1

are as follows.
  • client-6-x86
  • client-6-x86_64
  • client-7-x86_64
  • server-6-x86
  • server-6-x86_64
  • server-7-x86_64
  • workstation-6-x86
  • workstation-6-x86_64
  • workstation-7-x86_64
  • server-6-s390x
  • server-7-s390x
  • server-7-ppc64le
  • server-7-ppc64be

Entitlement certificates are placed in the correct folders

Ensure that the certificates are in the correct folders. Follow these guidelines to avoid errors.

  • The following path is the relative path where the rootCertDir is located. This can be left at its default value ("certs") in the plugin.ini
    ^^^^^^^^^^^^^^^​^^^^^^^^^
rootCertDir = certs
^^^^^^^^^^^^^^^​^^^^^^^^^
  • The "certs" folder must only contain subfolders. For example, cert_set_1, cert_set_. Remove metadata files.
  • Within the "cert_set_1" folder only files ending with ".pem" are allowed. There can be any no. of Entitlement Certificates in "cert_set_1". For example, 443229635427054308.pem. Only Entitlement Certificates with the new format are allowed.
    Note: Earlier versions of the RHSM subscription interface had an option to download the system identity certificate. This is no longer the case with the current RHSM subscription interface version. The System Identity Certificate is no longer required from v1.0.2.0 of the RHSM download plug-in and RHSM download cacher.
  • If you have more than one set of certificates, ensure that only one set of certificates go to one folder.

The version of RHSM entitlement certificates have the correct format

When creating RHSM certificates in the RHSM customer portal, and you are at the step in which you need to register a system, you must specify the Red Hat Enterprise Linux version.

To avoid errors, select version 7.2. Red Hat Enterprise Linux versions that are earlier than version 7.2 have a different entitlement certificate format that the RHSM download plug-in does not read. The new entitlement certificate format has "BEGIN ENTITLEMENT DATA" in the .pem file.

To verify the format version of the entitlement format, do the following steps:
  1. Open the .pem file in a text editor.
  2. Search for "BEGIN ENTITLEMENT DATA". Only the new entitlement format will have this; neither the old entitlement certificate format and the system identify certificate will have "BEGIN ENTITLEMENT DATA".
Note: Ensure that the Entitlement Certificates with the old format is deleted from <BES_Server>\DownloadPlugins\RHSMProtocol\certs\cert_set_1.

The entitlement certificates are active and have not expired

Certificates usually expire in 1 year to a few years. There have been some cases where patch deployment would fail one day after the entitlement certificates were created. The entitlement certificates were found to have an expiry date set 1 day after they were created. There are several ways to check the expiry date of the certificates.
  • Through a Red Hat machine.
  • Through the Red Hat portal. You must access the account in https://access.​redhat.com that generated the entitlement certificates.
  • Through OpenSSL
If the subscription is expired, you must generate or attach a new subscription (entitlement) to the entitlement certificate. Regenerate the identity certificate if it is expired. Place the downloaded certificates in the correct folder.
Verifying the correct subscription (certificate) attachments using a Red Hat machine

From the command line in a Red Hat machine, run > rct cat-cert <entitlement cert> > output.txt to print the certs metadata to the ouput.txt file. Repeat this for each Entitlement Certificate and the System Identity Certificate using a different output.txt filename. Open the file in a text editor and the certs expiry date will be in the End Date field End Date: 2018-05-25 12:50:11+00:00.

Verifying the correct subscription (certificate) attachments through the Red Hat portal
  1. Log in to https://access.​redhat.com.
  2. Go to https://access.​redhat.com/mana​gement/consumer​s?type=system
  3. Click the system that you previously created. A list of Entitlement Certificates displays.
  4. For each Entitlement Certificate, click View. Go to the Order Info tab. In the End Date column, verify that the subscription are not expired.
  5. Click Back in the browser and repeat steps 4 to 6 for each Attached Subscription in your system.
Verifying the correct subscription (certificate) attachments through OpenSSL

If you are able to use openssl, open a command line at this folder: \DownloadPlugins\RHSMProtocol\certs\cert_set_1

Use this command and replace the name of the “.pem" file: 

$ openssl x509 -enddate -noout -in 7a8337a5-eb47-4a52-a161-9635d5691996.pem
This results to the expiry date of the certificate. For example,
notAfter=Jan 10 15:19:14 2018 GMT

Entitlement certificates with the attached subscription (entitlement) with Name: Red Hat Enterprise Linux for Virtual Datacenters has been known to stop working after 1 day. If you are having issues with patch deployment after 1 day, we suggest avoiding this subscription and using a non-Virtual Datacenters subscription like Red Hat Enterprise Linux 7 Server (RPMs) instead.

Entitlement Certificates have the correct Subscriptions (Entitlements) attached

There are two ways to verify if the correct subscription (entitlement) is attached to the entitlement certificate. This is needed to get access to the required repositories. The first method requires having a Red Hat machine. The second method requires access to the account in https://access.​redhat.com that generated the entitlement certificates.
Verifying the correct subscription (certificate) attachments using a Red Hat machine

From the command line in a Red Hat machine, run > rct cat-cert <entitlement cert>. This displays the entitlement certificate metadata, including the expiry date of the certificate and the repositories that the certificate can access.

In the following example, Certificate: End Date: 2017-01-17 13:30:47+00:00 shows the expiry date. Content: Name: Red Hat Enterprise Linux 7 Server (RPMs) lists the repositories that the certificate it can access.

===============​===============​============

Certificate:

Path: 7a85f98153c2eb9​50153c73d2fb159​e5.pem

Version: 3.2

Serial: 368971143702890​3897

Start Date: 2016-03-31 04:00:00+00:00

End Date: 2017-01-17 13:30:47+00:00

Content:

Type: yum

Name: Red Hat Enterprise Linux 7 Server (RPMs)

Label: rhel-7-server-r​pmsVendor: Red Hat

URL: /content/dist/r​hel/server/7/$r​eleasever/$base​arch/os

GPG: file:///etc/pki​/rpm-gpg/RPM-GP​G-KEY-redhat-re​lease

Enabled: True

Expires: 86400

Required Tags: rhel-7-server

Arches: x86_64

===============​===============​============

You might need the following base repositories, depending on the endpoint that you deploy patches to.
  • Red Hat Enterprise Linux 6 Desktop (RPMs)
  • Red Hat Enterprise Linux 6 Workstation (RPMs)
  • Red Hat Enterprise Linux 6 Server (RPMs)
  • Red Hat Enterprise Linux 7 Desktop (RPMs)
  • Red Hat Enterprise Linux 7 Server (RPMs)
  • Red Hat Enterprise Linux 7 Workstation (RPMs)
If the base repository name of the RHEL version that you deploy patches for was not found in all the metadata of the entitlement certificates, it means that the required subscriptions were not attached. Entitlement certificates with the attached subscription (entitlement) with Name: Red Hat Enterprise Linux for Virtual Datacenters has been known to stop working after 1 day. If you are having issues with patch deployment after 1 day, we suggest avoiding this subscription and using a non-Virtual Datacenters subscription like Red Hat Enterprise Linux 7 Server (RPMs) instead.
Verifying the correct subscription (certificate) attachments by accessing the account on https://access.​redhat.com that generated the entitlement certificates
  1. Log in to https://access.​redhat.com.
  2. Go to https://access.​redhat.com/mana​gement/consumer​s?type=system
  3. Click the system you previously created. A list of list of attached subscriptions displays.
  4. For each subscription, in the Entitlement Certificate column, click View > Content Sets > Export All as CSV.
  5. Click Back in the browser and repeat Step 4 for each Attached Subscription in your system.
  6. Open each export.CSV that was downloaded from RedHat. Under the Name column, search for the Base repository name of the repositories that you need access to for patch deployment.
You might need the following base repository names, depending on the endpoint that you deploy patches to.
  • Red Hat Enterprise Linux 6 Desktop (RPMs)
  • Red Hat Enterprise Linux 6 Workstation (RPMs)
  • Red Hat Enterprise Linux 6 Server (RPMs)
  • Red Hat Enterprise Linux 7 Desktop (RPMs)
  • Red Hat Enterprise Linux 7 Server (RPMs)
  • Red Hat Enterprise Linux 7 Workstation (RPMs)
If the base repository name of the RHEL version that you deploy patches for was not found in all the export.csv, it means that the required subscriptions were not attached.

Entitlement certificate can access the Red Hat base repositories

Before you begin, ensure that Check 1 (Ensure that the entitlement certificates have the correct format) and 2 (Ensure that the entitlement certificates are placed in the correct folders) are completed before following the steps in this check.You can run commands that will help identify Red Hat repository access. The RHSMPlugin.exe uses the entitlement certificate to do the following:
  • Test the access to the Red Hat base repositories.
  • Determine if the proper subscriptions have been attached to the entitlement certificate.
Run the following:
  • For RHSMPlugin.exe (v1.0.0.2 and later): >>>RHSMPlugin.e​xe --check-baserep​os
  • For RHSMDownloadCac​her.exe (v1.0.0.2 and later) : >>> RHSMDownloadCac​her.exe --rootCertDir certs check-baserepos
This will test all base repos in the “primaryRepoLis​tFile" as specified in the plugin.ini. Depending on network conditions, this should take around 10 to 60 seconds. After the commands are run, the results are printed to the console as a “Base Repos Test Summary", and to the logs which will state which base repositories the certificates are able to access.
There are several possible reasons why the Entitlement Certificate are not able to access the repositories:
  • The certificates have expired. To remedy this, see CHECK 5: Entitlement Certificates and System Identity Certificate are not expired.
  • The required subscription were not properly attached when the system was registered through Red Hat Subscription Management portal. To remedy this, see CHECK 4: Entitlement Certificates have the correct Subscriptions (Entitlements) attached.
  • The network or proxy is blocking RHSMPlugin.exe from accessing the repositories. Check that your network firewall or proxy is not blocking the RHSMPlugin.exe. If the problem persists, you might need to contact Support.

Error messages in RHSMPlugin.log

The RHSMPlugin.log is located in <BES_Server>\DownloadPlugins\RHSMProtocol\logs.

You might encounter the following error in the log:

ERROR : All Key and Cert pairs in 'rootCertDir' cannot access: https://cdn.redhat.com/content/dist/rhel/client/7/7Client/x86_64/os/repodata/repomd.xml

This error message indicates that RHSMplugin was not able to access RedHat’s Client RHELClient 7 Repo (“rhel/client/7/7Client/x86_64/os").

This happens when the same package is found in multiple repositories. This will prompt the RHSM download plug-in to access all the repositories where the package is located. When the download plug-in tries to access a repository that it does not have access to and when there is a lack of entitlement of the Entitlement Certificate, it will the indicate the error in the log.

If you do not need to deploy patches to any endpoints, as for example, RHEL Client 7 machines

If you do need to deploy patches to any such endpoints, you may safely ignore this message. This error might be due to the client script nohash limitation which the RHSM download plug-in cannot avoid. For more information, see: https://developer.bigfix.com/action-script/reference/download/add-nohash-prefetch-item.html.

If the error message is associated with a repository that you need for your patch deployment, for example if you have a RHEL 7 Server endpoint that requires patching
This might be caused by any of the following reasons:
  • The certificates have expired or were revoked. To remediate this, see the steps in "Ensure that entitlement certificates are not expired."
  • The required subscriptions were not properly attached when registering the system through RedHat. To remediate this, see the steps in "Entitlement certificates are not able to access the repositories."
  • The network or proxy is blocking the RHSMPlugin.exe from accessing the repositories. Check that your network firewall or proxy is not blocking the RHSMPlugin.exe. If the problem persists, you might need to contact Support.
.
1 primaryRepoListFile