All Apps and Add-ons

How to send metrics data from kafka to Splunk metrics index

Splunk Employee
Splunk Employee

I'm able to send metrics data from kafka to Splunk event index, any idea how to send metrics data to metrics index as well?

0 Karma

Splunk Employee
Splunk Employee

For Kafka connect to ingest metrics data - we need to use collectd as documented here:

Once collectd is installed, below is an example of a connector to send collectd metrics to a Splunk metrics index
The Splunk metrics index is optimized for ingesting and retrieving metrics. For more information, see the Metrics manual.

curl http://localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{
"name": "metrics-via-collectd",
"config": {
  "connector.class": "com.splunk.kafka.connect.SplunkSinkConnector",
  "tasks.max": "1",
  "splunk.sourcetypes": "collectd_http",
  "splunk.indexes": "kafka_metrics",
  "splunk.hec.uri": "https://localhost:8088",
  "splunk.hec.token": "727ac06e-5150-4f0f-a85d-c7b070176a2c",
  "splunk.hec.ack.enabled" : "false",
  "splunk.hec.ack.poll.interval" : "20",
  "splunk.hec.ack.poll.threads" : "2",
  "splunk.hec.event.timeout" : "120",
  "splunk.hec.ssl.validate.certs": "false",
  "splunk.hec.raw" : "true",
  "splunk.hec.raw.line.breaker" : "####"

Then, you can use below sample data to send to kafka.

bin/ --broker-list localhost:9092 --topic collectd

Then, check from Splunk metrics index by using below command to confirm metrics data is indexed correctly.

| mcatalog values(metric_name) WHERE index=kafka_metrics AND metric_name=protocols.protocol_counter.InOctets.value
0 Karma
Get Updates on the Splunk Community!

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...