How to extract custom dimension from metrics in Splunk App for Infrastructure during index time?

[vladislavplaksy]

We are using Splunk App for Infrastructure (SAI) to analyze hadoop cluster nodes using Apache Hadoop plugin for collectd from SignalFx.

We have tried to configure collectd to send metrics to Splunk using both write_http or write_splunk plugin.
We have faced a problem as extracting custom dimensions from metrics.

We not only need "static" dimension for all metrics which we can configure in collectd plugin:

<Plugin write_splunk>
  ...
  Dimension "key1:value1"
</Plugin>

but also would like to send "dynamic" dimension like: id, user, name, queue, state and some other from Cluster Applications API, all together more than 10 dimensions.

If we use write_http plugin we have a problem to extract dimension as described in another question: How to extract custom dimensions from plugin_instance when we are using collectd?

When we are using write_splunk plugin our metric_name looks like this:

apache_hadoop.[id:application_1555643048019_55088,user:hive,queue:root.project_name,state:FINISHED].gauge.hadoop.resource.manager.apps.allocatedMB

It looks like plugin_instance dimension from collectd plugin is added to the metric name. And it's similar to how statsd adds dimension to the metric_name.

We would like to extract dimension form metric like this:

app_id=application_1555643048019_55088
user=hive
queue=root.project_name
state=FINISHED

We have tried to extract dimension like described at Examples of configuring dimension extraction using configuration as following:

# props.conf.example
[em_metrics]
METRICS_PROTOCOL = statsd
STATSD-DIM-TRANSFORMS = user, queue, app_id, state

# transforms.conf.example
[statsd-dims:user]
REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])
REMOVE_DIMS_FROM_METRIC_NAME = true

...

And it's not working for us.

Can you give any advice how to extract custom dimension from metrics in Splunk App for Infrastructure during index time?

[vladislavplaksy]

@dagarwal_splunk @pwu_splunk
As far as I understand you are working in Splunk.

• Could you provide a link to write_splunk documentation.
• Or link to source code, in case this plugin is open source.
• Or redirect this question to Splunk App for Infrastructure developers or support team.

[dagarwal_splunk]

I think you already have what you need for write_splunk https://docs.splunk.com/Documentation/InfraApp/1.3.0/Admin/ManageAgents .

It is not Open Source yet.

This document might help you: https://docs.splunk.com/Documentation/Splunk/7.2.6/Data/Configureindex-timefieldextraction .

[dagarwal_splunk]

You can try something like this to extract dimensions:

props.conf add:
[em_metrics]
TRANSFORMS-mname=mname

Transforms.conf add:
[mname]
SOURCE_KEY = field:metric_name
REGEX = apache_hadoop\.\[id:([^,]+),user:([^,]+),queue:([^,]+),state:([^\]]+)\]\.\S+
FORMAT = id::$1 user::$2 queue::$3 state::$4
WRITE_META = true

Make adjustments as you need..

[pwu_splunk]

One issue I notice is that

"REGEX = (\Qu:\E(?.*?)[\Q,\E\Q]\E])"

doesn't look like a correct regex format in Splunk.

See: https://docs.splunk.com/Documentation/Splunk/7.2.6/Knowledge/AboutSplunkregularexpressions

Edit: Even looking at it as the "\Q\E" flavor of Regex, I don't see where the "u:" is coming from.

[vladislavplaksy]

@pwu_splunk
Thank you for your reply. It was misspelling here.
We are using shortcut name of dimensions as for example: user -> u, queue -> q, state -> s etc.
I didn't changed this in all places. I have updated my question.
Correct version

    # transforms.conf.example
    [statsd-dims:user]
    REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])

[pwu_splunk]

REGEX = (\Quser:\E(?<user>.*?)[\Q,\E\Q]\E])

I still suspect this format is incorrect. See the link on how to format Splunk regexes and the table inside on valid characters. I've only seen standard-form regexes in Splunk.

[dagarwal_splunk]

Did you try the other props/transforms above?