Splunk 7.0.0: How to get metrics in from collectd

[sureshr7]

Hi,

How to add a tag(region) to a collectd based metric from a host? For example if we have 2 regions (us-east,us-west), how to add this data from collectd to the Splunk metrics index?

Thanks
Suresh

[sureshr7]

Using the below code, write_http will add custom tag "meta":{"foo":"bar"}" to all json data. Now the challenge is recognizing this as dimension in splunk.

PostCacheChain "PostCache"

<Target "set">
  MetaData "foo" "bar"
</Target>

Target "write"

[lukeh]

Thanks, I'll have a crack and get back to you.

[theouhuios]

Any luck with this? Is there a way to add Tags via collectd and get splunk to understand them as dimensions?

[sureshr7]

I was able to add a custom meta tag using PostCacheChain from collectd. But still not able to read it via splunk.

[lukeh]

I would be happy to test this for you, could you please provide an example from collectd.conf ?
FYI: I am the author of Analytics for Linux which uses metrics from collectd:
https://splunkbase.splunk.com/app/3777/

[thaggie_splunk]

You should be able to add a region field as you push the measurements to splunk.
From: http://docs.splunk.com/Documentation/Splunk/7.0.0/Metrics/GetMetricsInCollectd

curl -k https://localhost:8088/services/collector/raw?sourcetype=collectd_http   \
-H "Authorization: Splunk <HEC_token>"                                      \
-d '[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]'

So each message is like:

{
  "values": [164.9196798931339196],
  "dstypes": ["derive"],
  "dsnames": ["value"],
  "time": 1505356687.894,
  "interval": 10.000,
  "host": "collectd",
  "plugin": "protocols",
  "plugin_instance": "IpExt",
  "type": "protocol_counter",
  "type_instance": "InOctets"
}

You should be able to add:

"region": "us-east"

[lukeh]

that will work for custom metrics, but not for native collectd metrics as the Collectd API does not have a concept of tags. It looks like the Collectd community is working towards a solution – but it still looks like a bandaid attempting to work around the fact that Collectd’s core has no concept of tags:
https://github.com/collectd/collectd/pull/1655