All Apps and Add-ons

How to send metrics data from kafka to Splunk metrics index

Splunk Employee
Splunk Employee

I'm able to send metrics data from kafka to Splunk event index, any idea how to send metrics data to metrics index as well?

0 Karma

Splunk Employee
Splunk Employee

For Kafka connect to ingest metrics data - we need to use collectd as documented here:

Once collectd is installed, below is an example of a connector to send collectd metrics to a Splunk metrics index
The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual.

curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{
"name": "metrics-via-collectd",
"config": {
  "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
  "tasks.max": "1",
  "splunk.sourcetypes": "collectd_http",
  "splunk.indexes": "kafka_metrics",
  "splunk.hec.uri": "https://localhost:8088",
  "splunk.hec.token": "727ac06e-5150-4f0f-a85d-c7b070176a2c",
  "splunk.hec.ack.enabled" : "false",
  "splunk.hec.ack.poll.interval" : "20",
  "splunk.hec.ack.poll.threads" : "2",
  "splunk.hec.event.timeout" : "120",
  "splunk.hec.ssl.validate.certs": "false",
  "splunk.hec.raw" : "true",
  "splunk.hec.raw.line.breaker" : "####"

Then, you can use below sample data to send to kafka.

bin/ --broker-list localhost:9092 --topic collectd

Then, check from Splunk metrics index by using below command to confirm metrics data is indexed correctly.

| mcatalog values(metric_name) WHERE index=kafka_metrics AND metric_name=protocols.protocol_counter.InOctets.value
0 Karma
Get Updates on the Splunk Community!

Platform Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestIntroducing Splunk Edge Processor, simplified data ...

Enterprise Security Content Updates (ESCU) - New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 3 releases of new content via the Enterprise ...

Thought Leaders are Validating Your Hard Work and Training Rigor

As a Splunk enthusiast and member of the Splunk Community, you are one of thousands who recognize the value of ...