Kafka authentication using Kerberos

Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name as the user running the client), so obtain or create these principals as needed. Then configure the JAAS configuration property for each client. Different clients within a JVM may run as different users by specifying different principals.

Before starting Kafka kerberos configuration with Journey:

From Kerberos hosted machine copy all the files available at location /var/kerberos/krb5kdc to Journey installation location <Journey-HOME>/Journey/Web/properties

From Kerberos hosted machine copy krb5.conf file available at location /etc to Journey installation location -> <Journey-HOME>/Journey/Web/properties
Note: Host name of kafka_Hosted machine is added on Unica application hosted machine and vice versa and make sure that Kafka communication working good between two machines.
  1. Enable the Kafka Security for Journey Engine and Journey WEB
    • For Journey Engine - Please replace the krb5 file location to the <JOURNEY_HOME>/Engine/journey_master_config.properties

      kafka.security.enabled=Y

      kafka.security.protocols.enabled=GSSAPI

    • For Journey WEB - Please replace the krb5 file location to the <JOURNEY_HOME>/WEB/properties/application.properties

      kafka.security.enabled=Y

      kafka.security.protocols.enabled=GSSAPI

  2. Make sure the keytabs configured in the JAAS configuration are readable by the operating system user who is starting kafka client.

    • For Journey Engine - Please replace the kafka_server_jaas.conf file location to the <JOURNEY_HOME>/Engine/journey_master_config.properties

      java.security.auth.login.config = <jass LOCATION> example - /etc/kafka_server_jaas.conf

    • For Journey WEB - Please replace the kafka_server_jaas.conf file location to the <JOURNEY_HOME>/WEB/properties/application.properties

      java.security.auth.login.config = <jass LOCATION> example - /etc/kafka_server_jaas.conf

  3. Pass the krb5 file locations as JVM parameters to each client JVM

    • For Journey Engine - Please replace the krb5 file location to the <JOURNEY_HOME>/Engine/journey_master_config.properties

      java.security.krb5.conf = <krb5 LOCATION> example - /etc/krb5.conf

    • For Journey WEB - Please replace the krb5 file location to the <JOURNEY_HOME>/WEB/properties/application.properties

      java.security.krb5.conf = <krb5 LOCATION> example - /etc/krb5.conf

  4. While configuring Kerberos with Kafka add/update below changes in below file:
    Location - /<Journey-HOME>/Journey/KafkaStandalone/config/server.properties
    sasl.enabled.mechanisms=GSSAPI
sasl.mechanism.inter.broker.protocol=GSSAPI
security.inter.broker.protocol=SASL_PLAINTEXT
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://<Kafka_Kerberos_principle_host_name>:9092
#E.g. advertised.listeners=SASL_PLAINTEXT://ip-10-10-10-100.ap-south-1.compute.internal.nonprod.hclpnp.com:9092
listener.name.sasl_plaintext.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true
keyTab="/var/kerberos/krb5kdc/kfkserver.keytab" principal="kafka/<Kafka_Kerberos_principle_host_name>";
sasl.kerberos.service.name=kafka
#e.g. keyTab="/var/kerberos/krb5kdc/kfkserver.keytab" principal="kafka/ip-10-10-10-100.ap-south-1.compute.internal.nonprod.hclpnp.com"; 
sasl.kerberos.service.name=kafka
    Note: This value will change as per your kafka principle 
    ip-10-10-10-100.ap-south-1.compute.internal.nonprod.hclpnp.com@HCLPNP.COM
  5. Add Kafka kerberos host name in Journey Engine and Journey WEB application.properties file 
    spring.kafka.bootstrap-servers=<Kafka_Kerbersos_Host_Name>:9092
  6. Create folders by name -> krb5.conf.d and log at location /<Journey-HOME>/Journey/Web/properties
  7. update file kafka_server_kerberos_jaas.conf OR equivalent Kafka Kerberos client conf available at location /<Journey-HOME>/Journey/Web/properties with applicable kerberos details and recent paths as per Journey installation location 
    keyTab="/var/kerberos/krb5kdc/zkserver.keytab"
principal="zookeeper/ip-10-10-10-10.ec2.internal.nonprod.hclpnp.com";
keyTab="/var/kerberos/krb5kdc/kfkserver.keytab"
principal="kafka/ip-10-10-10-10.ec2.internal.nonprod.hclpnp.com"
  8. update file krb5.conf available at location /<Journey-HOME>/Journey/Web/properties with applicable kerberos details and recent paths as per Journey installation location: 
    includedir /<Journey-HOME>/Journey/Web/properties/krb5.conf.d/
 [logging]
 default = FILE:/<Journey-HOME>/Journey/Web/properties/log/krb5libs.log
 kdc = FILE:/<Journey-HOME>/Journey/Web/properties/log/krb5kdc.log
 admin_server = FILE:/<Journey-HOME>/Journey/Web/properties/log/kadmind.log

JBOSS Setting for Kerberos Kafka Authentication

The below changes has been done in Jboss standalone.xml file to before start the Journey web:
  1. Go to <JBOSS_HOME>\standalone\configuration\
  2. Open standalone.xml
    1. Confirm the servlet-container node must have all the below values. If not please replace the node with below provided set of lines
      <servlet-container name="default" disable-caching-for-secured-pages="false">
<jsp-config/>
<websockets/>
</servlet-container>
    2. kafka_server_kerberos_jaas.conf and krb5.conf file path configure in standalone.xml as mention below
      <system-properties>
 <property name="java.security.auth.login.config" value="<Journey-HOME>/Journey/Web/properties/kafka_server_kerberos_jaas.conf"/>
 <property name="java.security.krb5.conf" value="<Journey-HOME>/Journey/Web/properties/krb5.conf"/>
 <property name="java.security.krb5.debug" value="true"/>
 <property name="java.security.disable.secdomain.option" value="true"/>
 </system-properties>
    3. Add the KafkaClient and KafkaServer conf in standalone.xml as shown below
    <security-domain name="KafkaClient" cache-type="default">
 <authentication>
 <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
 <module-option name="storeKey" value="true"/>
 <module-option name="useKeyTab" value="true"/>
 <module-option name="refreshKrb5Config" value="true"/>
 <module-option name="principal" value="<CLIENT-PRINCIPLE>"/> example - "kafka/ip-10-10-10-10.ec2.nonprod.hclpnp.com"
 <module-option name="keyTab" value="<Journey-HOME>/Journey/Web/properties/kfkserver.keytab"/>
 <module-option name="doNotPrompt" value="true"/>
 </login-module>
 </authentication>
</security-domain>
<security-domain name="KafkaServer" cache-type="default">
 <authentication>
 <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
 <module-option name="storeKey" value="true"/>
 <module-option name="useKeyTab" value="true"/>
 <module-option name="refreshKrb5Config" value="true"/>
 <module-option name="principal" value="<Server-PRINCIPLE>"/> example - "zookeeper/ip-10-10-10-10.ec2.nonprod.hclpnp.com"
 <module-option name="keyTab" value="<Journey-HOME>/Journey/Web/properties/zkserver.keytab"/>
 <module-option name="doNotPrompt" value="true"/>
 </login-module>
 </authentication>
</security-domain>
    Note: The values of the above mentioned parameter keyTab and principle must be same which has been generated at the time of Kerberos Server setup.
  3. After saving the above changes in Jboss <JBOSS_HOME>\standalone\configurationstandalone.xml, Journey Web can be started.After configuring Journey with Kafka Kerberos, on Journey while creating Rest Entry source if application gives error like below user needs to verify node configuration of Confirm the servlet-container node as below.

    Errors: Unable to fetch rest Api Or Unable to fetch Kafka

    • Confirm the servlet-container node must have all the below values. If not please replace the node with below provided set of lines at below location:
      <JBOSS_HOME>\standalone\configuration\standalone.xml
<servlet-container name="default" disable-caching-for-secured-pages="false">
<jsp-config/>
<websockets/>
</servlet-container>

WAS Setting for Kerberos Kafka Authentication

There are 2 files which needs to set in the environment of WebSphere.
  1. Set krb5 in System Env

    For Krb5 Server > Server Type > Select the server > Java and Process Management > Process Definition > Java Virtual Machine > Generic JVM arguments append the below mentioned argument in the Argument List

    -Djava.security.krb5.conf=<KRB File Location>\krb5.conf

  2. jass entries

    Go to <WebSphere Home>\AppServer\profiles\<Profile>\properties\wsjaas.conf

    Append the following entries into the file

    KafkaServer

    { com.ibm.security.auth.module.Krb5LoginModule required useKeytab="<Zookeper Key Tab Path>\zkserver.keytab" storeKey=true principal= <Zookeper Principle> eg- "zookeeper/ip-10-10-10-10.ap-south-1.compute.internal.nonprod.hclpnp.com" credsType = both; }

    KafkaClient

    { com.ibm.security.auth.module.Krb5LoginModule required useKeytab="<Kafka Server Kay Tab Path>\kfkserver.keytab" principal= <Kafka Principle> eg - "kafka/ip-10-10-10-10.ap-south-1.compute.internal.nonprod.hclpnp.com" credsType = both debug=true; }

    Restart Application Server.

Configuring Kafka Kerberose for Deliver Responses:

Login to Unica application and navigate to below location:
  1. Location > Settings > configuration > HCL Unica > Journey > Kafka > Configurations

    Set:

    KafkaBrokerURL:     <Priciple_Hostname>:9092
CommunicationMechanism:     GSSAPI
sasl.mechanism: GSSAPI
sasl.jass.config.location: <JOURNEY_HOME>/WEB/properties/application.properties/kafka_server_Kerberose_jaas.conf
java.security.krb5.conf.location: <JOURNEY_HOME>/WEB/properties/application.properties/krb5.conf
    .
  2. Location: Settings > configuration > HCL Unica > Deliver > Kafka > RCT

    Set:

    Set 
KafkaBrokerURL:     <Priciple_Hostname>:9092
CommunicationMechanism:     SASL_PLAINTEXT
sasl.mechanism: GSSAPI

Kafka Configuration for Configuring External resources in Journey (AssetPicker and Journey configuration)

Login to Unica application and navigate to below location:

Location > Settings > configuration > HCL Unica > Journey > Integration > dataSource > <System_Configuration Template Name> > Kafka Configurations
Bootstrap servers (comma separated list of hosts)          <Karberos_Kafka_Principle_Host:>7001 
(e.g. ip-10-10-10-100.nonprod.hclpnp.com:7001){}
Security protocol             SASL_PLAINTEXT
SASL mechanism             KERBEROS
Kerberos - Configuration file path (e.g. /etc/krb5.conf, C:/Windows/krb5.ini)               <JOURNEY_HOME>/WEB/properties/application.properties/krb5.conf
Kerberos - Keytab file path               <JOURNEY_HOME>/WEB/properties/kfkserver.keytab
Kerberos - Principal        kafka/<Karberos_Kafka_Principle_Host>@HCLPNP.COM
e.g. kafka/ip-10-10-10-100.nonprod.hclpnp.com@HCLPNP.COM 
Kerberos - Service Name              kafka