How to send metrics data from kafka to Splunk metrics index

[daniel_splunk]

I'm able to send metrics data from kafka to Splunk event index, any idea how to send metrics data to metrics index as well?

[daniel_splunk]

For Kafka connect to ingest metrics data - we need to use collectd as documented here:

https://www.splunk.com/blog/2018/04/25/unleashing-data-ingestion-from-apache-kafka.html

Once collectd is installed, below is an example of a connector to send collectd metrics to a Splunk metrics index
The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual.

curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{
"name": "metrics-via-collectd",
"config": {
  "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
  "tasks.max": "1",
  "topics":"collectd",
  "splunk.sourcetypes": "collectd_http",
  "splunk.indexes": "kafka_metrics",
  "splunk.hec.uri": "https://localhost:8088",
  "splunk.hec.token": "727ac06e-5150-4f0f-a85d-c7b070176a2c",
  "splunk.hec.ack.enabled" : "false",
  "splunk.hec.ack.poll.interval" : "20",
  "splunk.hec.ack.poll.threads" : "2",
  "splunk.hec.event.timeout" : "120",
  "splunk.hec.ssl.validate.certs": "false",
  "splunk.hec.raw" : "true",
  "splunk.hec.raw.line.breaker" : "####"
  }
}'

Then, you can use below sample data to send to kafka.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic collectd
[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]

Then, check from Splunk metrics index by using below command to confirm metrics data is indexed correctly.

| mcatalog values(metric_name) WHERE index=kafka_metrics AND metric_name=protocols.protocol_counter.InOctets.value