Hi,
In the last couple of days i worked on deployment of Prometheus server and agent for Kafka monitoring. In that purpose i will share with you the main points that you need to do in order to achieve this.
First thing to do is to use the prometheus and grafana modules that you will find at the following links:
https://forge.puppet.com/puppet/prometheus
https://forge.puppet.com/puppet/grafana
After these are imported in puppet you need to create the following puppet files:
grafana.pp
class profiles::grafana {
class { '::grafana':
cfg => {
app_mode => 'production',
server => {
http_port => 8080,
},
database => {
type => 'sqlite3',
host => '127.0.0.1:3306',
name => 'grafana',
user => 'root',
password => 'grafana',
},
users => {
allow_sign_up => false,
},
},
}
}puppetserver.pp
class profiles::prometheusserver {
$kafka_nodes=hiera(profiles::prometheusserver::nodes)
if $kafka_nodes {
class {'::prometheus':
global_config => { 'scrape_interval'=> '15s', 'evaluation_interval'=> '15s', 'external_labels'=> { 'monitor'=>'master'}},
rule_files => [ "/etc/prometheus/alert.rules" ],
scrape_configs => [ {'job_name'=>'prometheus','scrape_interval'=> '30s','scrape_timeout'=>'30s','static_configs'=> [{'targets'=>['localhost:9090'], 'labels'=> { 'alias'=>'Prometheus'}}]},{'job_name'=> kafka, 'scrape_interval'=> '10s', 'scrape_timeout'=> '10s', 'static_configs'=> [{'targets'=> $kafka_nodes }]}],
}
} else {
class {'::prometheus':
global_config => { 'scrape_interval'=> '15s', 'evaluation_interval'=> '15s', 'external_labels'=> { 'monitor'=>'master'}},
rule_files => [ "/etc/prometheus/alert.rules" ],
scrape_configs => [ {'job_name'=>'prometheus','scrape_interval'=> '30s','scrape_timeout'=>'30s','static_configs'=> [{'targets'=>['localhost:9090'], 'labels'=> { 'alias'=>'Prometheus'}}]}],
}
}
}prometheusnode.pp
class profiles_opqs::prometheusnode(
$jmxexporter_dir = hiera('jmxexporter::dir','/opt/jmxexporter'),
$jmxexporter_version = hiera('jmxexporter::version','0.9')
){
include ::prometheus::node_exporter
#validate_string($jmxexporter_dir)
file {"${jmxexporter_dir}":
ensure => 'directory',
}
file {"${jmxexporter_dir}/prometheus_config.yaml":
source => 'puppet:///modules/profiles/prometheus_config',
}
wget::fetch {"https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/${jmxexporter_version}/jmx_prometheus_javaagent-${jmxexporter_version}.jar":
destination => "${jmxexporter_dir}/",
cache_dir => '/tmp/',
timeout => 0,
verbose => false,
unless => "test -e ${jmxexporter_dir}/jmx_prometheus_javaagent-${jmxexporter_version}.jar",
}
}It is true that i used the wget module to take the JMX exporter so i must give you that as well
https://forge.puppet.com/leonardothibes/wget
As also required is the file to configure the jmx exporter configuration file in order to translate the JMX data provided by Kafka to fields to be imported in Prometheus
prometheus_config:
lowercaseOutputName: true
rules:
- pattern : kafka.cluster<type=(.+), name=(.+), topic=(.+), partition=(.+)><>Value
name: kafka_cluster_$1_$2
labels:
topic: "$3"
partition: "$4"
- pattern : kafka.log<type=Log, name=(.+), topic=(.+), partition=(.+)><>Value
name: kafka_log_$1
labels:
topic: "$2"
partition: "$3"
- pattern : kafka.controller<type=(.+), name=(.+)><>(Count|Value)
name: kafka_controller_$1_$2
- pattern : kafka.network<type=(.+), name=(.+)><>Value
name: kafka_network_$1_$2
- pattern : kafka.network<type=(.+), name=(.+)PerSec, request=(.+)><>Count
name: kafka_network_$1_$2_total
labels:
request: "$3"
- pattern : kafka.network<type=(.+), name=(\w+), networkProcessor=(.+)><>Count
name: kafka_network_$1_$2
labels:
request: "$3"
type: COUNTER
- pattern : kafka.network<type=(.+), name=(\w+), request=(\w+)><>Count
name: kafka_network_$1_$2
labels:
request: "$3"
- pattern : kafka.network<type=(.+), name=(\w+)><>Count
name: kafka_network_$1_$2
- pattern : kafka.server<type=(.+), name=(.+)PerSec\w*, topic=(.+)><>Count
name: kafka_server_$1_$2_total
labels:
topic: "$3"
- pattern : kafka.server<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_server_$1_$2_total
type: COUNTER
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern : kafka.server<type=(.+), name=(.+), topic=(.+), partition=(.*)><>(Count|Value)
name: kafka_server_$1_$2
labels:
topic: "$3"
partition: "$4"
- pattern : kafka.server<type=(.+), name=(.+), topic=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
topic: "$3"
type: COUNTER
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
broker: "$4:$5"
- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
- pattern : kafka.server<type=(.+), name=(.+)><>(Count|Value)
name: kafka_server_$1_$2
- pattern : kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_$1_$2_$3_total
- pattern : kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, topic=(.+)><>Count
name: kafka_$1_$2_$3_total
labels:
topic: "$4"
type: COUNTER
- pattern : kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, topic=(.+), partition=(.+)><>Count
name: kafka_$1_$2_$3_total
labels:
topic: "$4"
partition: "$5"
type: COUNTER
- pattern : kafka.(\w+)<type=(.+), name=(.+)><>(Count|Value)
name: kafka_$1_$2_$3_$4
type: COUNTER
- pattern : kafka.(\w+)<type=(.+), name=(.+), (\w+)=(.+)><>(Count|Value)
name: kafka_$1_$2_$3_$6
labels:
"$4": "$5"
Ok, so in order to put this together, we will use plain old hiera :). For the server on which you want to configure prometheus server you will need to create a role or just put it in the fqdn.yaml that looks like this:
prometheus.yaml
---
classes:
- 'profiles::prometheusserver'
- 'profiles::grafana'
alertrules:
-
name: 'InstanceDown'
condition: 'up == 0'
timeduration: '5m'
labels:
-
name: 'severity'
content: 'critical'
annotations:
-
name: 'summary'
content: 'Instance {{ $labels.instance }} down'
-
name: 'description'
content: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
This is as default installation, because it’s a “role”, on each prometheus host i also created a specific fqdn.yaml file to specify in order to tell what nodes should be checked for exposed metrics. Here is an example:
---
profiles::prometheusserver::nodes:
- 'kafka0:7071'
- 'kafka1:7071'
- 'kafka2:7071'
The three nodes are as an example, you can put all the nodes on which you include the prometheus node class.
Let me show you how this should also look:
--- classes: - 'profiles::prometheusnode' profiles::kafka::jolokia: '-javaagent:/usr/share/java/jolokia-jvm-agent.jar -javaagent:/opt/jmxexporter/jmx_prometheus_javaagent-0.9.jar=7071:/opt/jmxexporter/prometheus_config.yaml
Now i need to explain that jolokia variable, right? Yeah, it’s pretty straight forward. The kafka installation was already wrote, and it included the jolokia agent and our broker definition block looks like this:
class { '::kafka::broker':
config => $broker_config,
opts => hiera('profiles::kafka::jolokia', '-javaagent:/usr/share/java/jolokia-jvm-agent.jar'),
heap_opts => "-Xmx${jvm_heap_size}M -Xms${jvm_heap_size}M",
}
}So i needed to puth the jmx exporter agent beside jolokia on kafka startup, and when this will be deployed you will see the jmxexporter started as agent. Anyhow, when all is deployed you will have a prometheus config that should look like:
---
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: master
rule_files:
- /etc/prometheus/alert.rules
scrape_configs:
- job_name: prometheus
scrape_interval: 30s
scrape_timeout: 30s
static_configs:
- targets:
- localhost:9090
labels:
alias: Prometheus
- job_name: kafka
scrape_interval: 10s
scrape_timeout: 10s
static_configs:
- targets:
- kafka0:7071
- kafka1:7071
- kafka2:7071
You can also see the nodes at Status -> Targets from the menu, and yeah, all the metrics are available by node at http://[kafka-node]:7071/metrics.
I think this should be it, i don’t know i covered everything and there are a lot of details related to our custom installation but at least i managed to provide so details related to it. The article that helped me very much to do is can be visited here
https://www.robustperception.io/monitoring-kafka-with-prometheus/
Cheers!