Splunk Search

Dynamic field extraction name

phemmer
Path Finder

I am trying to extract some fields from some reporting data. The reporting data has a field name, and then a bunch of metrics related to that field. For example a log entry might look like

report=memory ram_used=123mb;500;600;0;700 swap_used=456mb;1100;1200;0;1300

Each of the stats is "{CURRENT}[UNIT];{WARNING};{CRITICAL};{MINIMUM};{MAXIMUM}". I want to end up with fields such as ram_used_current=123, ram_used_unit=mb, swap_used_max=1300, etc.

My first thought was to solve this by doing a transform such as

[reporting]
REGEX = ([^\s=]+)=([0-9\.\-\+_]+)([a-zA-Z]*);([^\s;]*);([^\s;]*);([^\s;]*);([^\s;]*)
FORMAT = $1_current::$2 $1_unit::3 $1_warn::$4 $1_crit::$5 $1_min::$6 $1_max::$7

But this doesn't appear to work. It looks like it's just setting fields such as current instead of the desired ram_used_current.

I know in the transforms.conf documentation it says you cannot concatenate fields, but I'm assuming this means values, and not the field names. As you can create whatever field names you want by doing named regexes, so this seems to indicate the engine is capable of handling it.

kristian_kolb
Ultra Champion

You could make that as an EXTRACT in props.conf only. It will not alter the events, like a transform would do. It will work on already indexed data.

Something like this (not extracting the unit);

props.conf

[your sourcetype]
EXTRACT-YYY = ram_used=(?<ram_used_current>\d+)\w+;(?<ram_used_warn>\d+);(?<ram_used_crit>\d+);(?<ram_used_min>\d+);(?<ram_used_max>\d+);
EXTRACT-XXX = swap_used=(?<swap_used_current>\d+) etc etc etc

/K

0 Karma

phemmer
Path Finder

This requires me to write a regex for every single group of statistics we have. There are hundreds of them, they change frequently, and some of them are even dynamic (they come and go automatically).

0 Karma
Get Updates on the Splunk Community!

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...