topic Re: Dynamic field extraction name in Splunk Search

Dynamic field extraction name

phemmer — Thu, 18 Apr 2013 03:07:22 GMT

I am trying to extract some fields from some reporting data. The reporting data has a field name, and then a bunch of metrics related to that field. For example a log entry might look like

report=memory ram_used=123mb;500;600;0;700 swap_used=456mb;1100;1200;0;1300

Each of the stats is "{CURRENT}[UNIT];{WARNING};{CRITICAL};{MINIMUM};{MAXIMUM}". I want to end up with fields such as ram_used_current=123, ram_used_unit=mb, swap_used_max=1300, etc.

My first thought was to solve this by doing a transform such as

[reporting]
REGEX = ([^\s=]+)=([0-9\.\-\+_]+)([a-zA-Z]*);([^\s;]*);([^\s;]*);([^\s;]*);([^\s;]*)
FORMAT = $1_current::$2 $1_unit::3 $1_warn::$4 $1_crit::$5 $1_min::$6 $1_max::$7

But this doesn't appear to work. It looks like it's just setting fields such as current instead of the desired ram_used_current.

I know in the transforms.conf documentation it says you cannot concatenate fields, but I'm assuming this means values, and not the field names. As you can create whatever field names you want by doing named regexes, so this seems to indicate the engine is capable of handling it.

Re: Dynamic field extraction name

kristian_kolb — Thu, 18 Apr 2013 10:20:41 GMT

You could make that as an EXTRACT in props.conf only. It will not alter the events, like a transform would do. It will work on already indexed data.

Something like this (not extracting the unit);

props.conf

[your sourcetype]
EXTRACT-YYY = ram_used=(?<ram_used_current>\d+)\w+;(?<ram_used_warn>\d+);(?<ram_used_crit>\d+);(?<ram_used_min>\d+);(?<ram_used_max>\d+);
EXTRACT-XXX = swap_used=(?<swap_used_current>\d+) etc etc etc

Re: Dynamic field extraction name

phemmer — Thu, 18 Apr 2013 12:05:24 GMT

This requires me to write a regex for every single group of statistics we have. There are hundreds of them, they change frequently, and some of them are even dynamic (they come and go automatically).