I'm able to send metrics data from kafka to Splunk event index, any idea how to send metrics data to metrics index as well?
For Kafka connect to ingest metrics data - we need to use collectd as documented here:
https://www.splunk.com/blog/2018/04/25/unleashing-data-ingestion-from-apache-kafka.html
Once collectd is installed, below is an example of a connector to send collectd metrics to a Splunk metrics index
The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual.
curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{
"name": "metrics-via-collectd",
"config": {
"connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
"tasks.max": "1",
"topics":"collectd",
"splunk.sourcetypes": "collectd_http",
"splunk.indexes": "kafka_metrics",
"splunk.hec.uri": "https://localhost:8088",
"splunk.hec.token": "727ac06e-5150-4f0f-a85d-c7b070176a2c",
"splunk.hec.ack.enabled" : "false",
"splunk.hec.ack.poll.interval" : "20",
"splunk.hec.ack.poll.threads" : "2",
"splunk.hec.event.timeout" : "120",
"splunk.hec.ssl.validate.certs": "false",
"splunk.hec.raw" : "true",
"splunk.hec.raw.line.breaker" : "####"
}
}'
Then, you can use below sample data to send to kafka.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic collectd
[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]
Then, check from Splunk metrics index by using below command to confirm metrics data is indexed correctly.
| mcatalog values(metric_name) WHERE index=kafka_metrics AND metric_name=protocols.protocol_counter.InOctets.value