We are using Splunk App for Infrastructure (SAI) to analyze hadoop cluster nodes using Apache Hadoop plugin for collectd from SignalFx.
We have tried to configure collectd to send metrics to Splunk using both write_http or write_splunk plugin.
We have faced a problem as extracting custom dimension s from metrics.
We not only need "static" dimension for all metrics which we can configure in collectd plugin:
<Plugin write_splunk>
...
Dimension "key1:value1"
</Plugin>
but also would like to send "dynamic" dimension like: id , user , name , queue , state and some other from Cluster Applications API, all together more than 10 dimensions.
If we use write_http plugin we have a problem to extract dimension as described in another question: How to extract custom dimensions from plugin_instance when we are using collectd?
When we are using write_splunk plugin our metric_name looks like this:
apache_hadoop.[id:application_1555643048019_55088,user:hive,queue:root.project_name,state:FINISHED].gauge.hadoop.resource.manager.apps.allocatedMB
It looks like plugin_instance dimension from collectd plugin is added to the metric name. And it's similar to how statsd adds dimension to the metric_name .
We would like to extract dimension form metric like this:
app_id=application_1555643048019_55088
user=hive
queue=root.project_name
state=FINISHED
We have tried to extract dimension like described at Examples of configuring dimension extraction using configuration as following:
# props.conf.example
[em_metrics]
METRICS_PROTOCOL = statsd
STATSD-DIM-TRANSFORMS = user, queue, app_id, state
# transforms.conf.example
[statsd-dims:user]
REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])
REMOVE_DIMS_FROM_METRIC_NAME = true
...
And it's not working for us.
Can you give any advice how to extract custom dimension from metrics in Splunk App for Infrastructure during index time?
... View more