Splunk Search

Dynamic field extraction name

phemmer
Path Finder

I am trying to extract some fields from some reporting data. The reporting data has a field name, and then a bunch of metrics related to that field. For example a log entry might look like

report=memory ram_used=123mb;500;600;0;700 swap_used=456mb;1100;1200;0;1300

Each of the stats is "{CURRENT}[UNIT];{WARNING};{CRITICAL};{MINIMUM};{MAXIMUM}". I want to end up with fields such as ram_used_current=123, ram_used_unit=mb, swap_used_max=1300, etc.

My first thought was to solve this by doing a transform such as

[reporting]
REGEX = ([^\s=]+)=([0-9\.\-\+_]+)([a-zA-Z]*);([^\s;]*);([^\s;]*);([^\s;]*);([^\s;]*)
FORMAT = $1_current::$2 $1_unit::3 $1_warn::$4 $1_crit::$5 $1_min::$6 $1_max::$7

But this doesn't appear to work. It looks like it's just setting fields such as current instead of the desired ram_used_current.

I know in the transforms.conf documentation it says you cannot concatenate fields, but I'm assuming this means values, and not the field names. As you can create whatever field names you want by doing named regexes, so this seems to indicate the engine is capable of handling it.

kristian_kolb
Ultra Champion

You could make that as an EXTRACT in props.conf only. It will not alter the events, like a transform would do. It will work on already indexed data.

Something like this (not extracting the unit);

props.conf

[your sourcetype]
EXTRACT-YYY = ram_used=(?<ram_used_current>\d+)\w+;(?<ram_used_warn>\d+);(?<ram_used_crit>\d+);(?<ram_used_min>\d+);(?<ram_used_max>\d+);
EXTRACT-XXX = swap_used=(?<swap_used_current>\d+) etc etc etc

/K

0 Karma

phemmer
Path Finder

This requires me to write a regex for every single group of statistics we have. There are hundreds of them, they change frequently, and some of them are even dynamic (they come and go automatically).

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...