We have tried to configure collectd to send metrics to Splunk using both
We have faced a problem as extracting custom
dimensions from metrics.
We not only need "static" dimension for all metrics which we can configure in collectd plugin:
<Plugin write_splunk> ... Dimension "key1:value1" </Plugin>
but also would like to send "dynamic" dimension like:
state and some other from Cluster Applications API, all together more than 10 dimensions.
If we use
write_http plugin we have a problem to extract dimension as described in another question: How to extract custom dimensions from plugin_instance when we are using collectd?
When we are using
write_splunk plugin our
metric_name looks like this:
It looks like
plugin_instance dimension from collectd plugin is added to the metric name. And it's similar to how statsd adds dimension to the
We would like to extract dimension form metric like this:
app_id=application_1555643048019_55088 user=hive queue=root.project_name state=FINISHED
We have tried to extract dimension like described at Examples of configuring dimension extraction using configuration as following:
# props.conf.example [em_metrics] METRICS_PROTOCOL = statsd STATSD-DIM-TRANSFORMS = user, queue, app_id, state # transforms.conf.example [statsd-dims:user] REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E]) REMOVE_DIMS_FROM_METRIC_NAME = true ...
And it's not working for us.
Can you give any advice how to extract custom dimension from metrics in Splunk App for Infrastructure during index time?
I think you already have what you need for write_splunk https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/ManageAgents .
It is not Open Source yet.
This document might help you: https://docs.splunk.com/Documentation/Splunk/7.2.6/Data/Configureindex-timefieldextraction .
You can try something like this to extract dimensions:
SOURCE_KEY = field:metric_name
REGEX = apache_hadoop\.\[id:([^,]+),user:([^,]+),queue:([^,]+),state:([^\]]+)\]\.\S+
FORMAT = id::$1 user::$2 queue::$3 state::$4
WRITE_META = true
Make adjustments as you need..
One issue I notice is that
"REGEX = (\Qu:\E(?.*?)[\Q,\E\Q]\E])"
doesn't look like a correct regex format in Splunk.
Edit: Even looking at it as the "\Q\E" flavor of Regex, I don't see where the "u:" is coming from.
Thank you for your reply. It was misspelling here.
We are using shortcut name of dimensions as for example: user -> u, queue -> q, state -> s etc.
I didn't changed this in all places. I have updated my question.
# transforms.conf.example [statsd-dims:user] REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])
REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])
I still suspect this format is incorrect. See the link on how to format Splunk regexes and the table inside on valid characters. I've only seen standard-form regexes in Splunk.