Kafka authentication using SSL

If you are using your organization's Kafka instance, you can use certificates configured for that Kafka instance. You are not required to generate SSL key and certificates and obtain the client certificates to configure in Journey application properties.

If you do not have the certificates, you can generate self-signed certificate authority (CA), which is simply a public-private key pair and certificate.

You must add the same CA certificate to each Kafka client and broker’s trust store.

Generate SSL key and certificate for each Kafka broker

To generate self-signed certificates for Kafka server, complete the following steps.

Prerequisites

  • You must have Java keytool and OpenSSL to generate certificates and trust store.
  • Optionally, you can use any SSL certificate generation utility instead of OpenSSL.
  1. To deploy SSL, generate the key and the certificate for each machine in the cluster. Generate the key into a temporary keystore initially so that you can export and sign it later with CA.
    keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey
    • keystore: The keystore file that stores the certificate. The keystore file contains the private key of the certificate; therefore, it needs to be kept safely.
    • validity: The valid time of the certificate in days.
  2. Create your own CA (certificate authority)

    openssl req -new -x509 -keyout ca-key -out ca-cert -days 365

    The generated CA is simply a public-private key pair and certificate, and it is intended to sign other certificates.

  3. Add the generated CA to the clients’ trust store so that the clients can trust this CA.
    • keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert
    • keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
  4. Sign all certificates in the keystore with the CA generated.
    1. Export the certificate from the keystore:

      keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file

  5. Sign it with CA.

    openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:<password>

  6. Import both the certificates of the CA and the signed certificate into the keystore.

    keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert

    keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed

  7. Create client keystore and import both certificates of the CA and signed certificates to client keystore. These client certificates will be used in application properties.

    keytool -keystore kafka.client.keystore.jks -alias localhost -validity 365 -genkey

    keytool -keystore kafka.client.keystore.jks -alias localhost -certreq -file cert-file

    openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:<password>

    keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert

    keytool -keystore kafka.client.keystore.jks -alias localhost -import -file cert-signed