Hi,
Do not make my mistake from the last couple of weeks trying to connect to a “secured” kafka cluster using TLS. I wrote following article http://log-it.tech/2017/07/27/configure-kafka-truststore-keystore-using-puppet/ some time ago, and i know that it’s far from bullet proof but it does the job.
Now let’s get to the subject, if you want to connect to the node once this is activated you can not use localhost anymore. And the way i figured it out is by trying to test the port using openssl command.
The config in server.properties is
'listeners' => "PLAINTEXT://${::fqdn}:9092,SSL://${::fqdn}:9093", #both listeners are enabled
'advertised.listeners' => "PLAINTEXT://${::fqdn}:9092,SSL://${::fqdn}:9093",
So, please keep in mind that it’s configured to listen on FQDN, so normally the external interface is the target not the loopback adapter.
Now if you try to test it using localhost you will surely get this output:
/opt/kafka/bin# openssl s_client -debug -connect localhost:9093 -tls1 connect: Connection refused connect:errno=111
Do not try to check if the firewall or port it’s opened. You can easily check that using iptables -L or netstat -tulpen | grep 9093. The problem is that instead of localhost you should be using FQDN like openssl s_client -debug -connect ${fqdn}:9093 -tls1 and you will see a lot of keys/certificates.
Now, if you want for example to use the standard .sh scripts that are delivered with kafka installation, you should created a file called config.properties (for example) and pass it as parameter. In case zookeeper connect (with the –zookeeper parameter) this is not needed but if you want to start a console consumer or producer, or you want to check the consumer groups, this will be needed. Let me just give you an example:
/opt/kafka/bin# ./kafka-consumer-groups.sh --command-config /root/config.properties --bootstrap-server ${fqdn}:9093 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
console-consumer-30514
KMOffsetCache-kafka2
KMOffsetCache-kafka0
KMOffsetCache-kafka1
Oterwise, it will not work. And my config file looks like this:
security.protocol=SSL ssl.truststore.location=/home/kafka/kafka.client.truststore.jks ssl.truststore.password=password ssl.keystore.location=/home/kafka/kafka.client.keystore.jks ssl.keystore.password=password ssl.key.password=password
I can not give you all the details to all the commands but at least i am confident i put you on the right track.
Cheers