Splunk Search

Extracting multiple similar values from a multi-line event

mikelanghorst
Motivator

I've got a rather tricky (at least for me) data set that I'd like to extract values from. For this example text

`

Elapsed Time Unti Event Locked Display :: Longest 5 Entries:
Elapsed Time Unti Event Locked Display :: [Max(0.361), Avg(0.180), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_RTD
Elapsed Time Unti Event Locked Display :: [Max(0.001), Avg(0.000), Min(0.000)] sec(s) # Total: 00007 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_WFC_STATUS
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00001 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_RTPD
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_REVIEW_PERIOD
>>> Eventing Processing Time :: Longest 5 Entries:
Eventing Processing Time :: [Max(0.421), Avg(0.308), Min(0.194)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD
Eventing Processing Time :: [Max(0.198), Avg(0.194), Min(0.190)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_REVIEW_PERIOD
Eventing Processing Time :: [Max(0.149), Avg(0.142), Min(0.134)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_RTD
Eventing Processing Time :: [Max(0.039), Avg(0.039), Min(0.039)] sec(s) # Total: 00001 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_RTPD
Eventing Processing Time :: [Max(0.017), Avg(0.011), Min(0.000)] sec(s) # Total: 00007 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_WFC_STATUS
User Sessions :: Number of sessions 1
`

I need to extract out the Max,Avg,Min values for each MenuId: in Eventing Processing Time. Max isn't a multivalue field, but rather only relevent to the MenuId in the same line. So I'd need something like RT_RTD_REVIEW_PERIOD-Max and RT_RTD_REVIEW_PERIOD-Avg.

I'm not sure how to do this other than simple brute forcing with multiple regexes.

Tags (1)
0 Karma
1 Solution

emiller42
Motivator

I believe the following would work, but only as an index-time extraction. (Not tested)

in Transforms.conf

[stanza_name]
REPEAT_MATCH = true
REGEX = Elapsed\sProcessing\sTime\s::\s\[\w+\(([^\)]+)\),\s\w+\(([^\)]+)\),\s\w+\(([^\)]+)\)\].+?#\s\w+:\s(\d+)\s+\w+:\s[\w\.]+-([\w_]+)
FORMAT = $5-Max::$1 $5-Avg::$2 $5-Min::$3 $5-Total::$4

My regex may not be the most clean/efficient, but it appears to capture everything correctly on regexr.

So from:

Eventing Processing Time :: [Max(0.421), Avg(0.308), Min(0.194)] sec(s)      # Total: 00002      MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD

it will extract:

$1 = 0.421
$2 = 0.308
$3 = 0.194
$4 = 00002
$5 = RT_RTD_REVIEW_PERIOD

So $5-Max::$1 should become 'RT_RTD_REVIEW_PERIOD-Max = 0.421'

According to the documentation on transforms.conf, you can only do concatenated fields with index-time extractions.

View solution in original post

emiller42
Motivator

I believe the following would work, but only as an index-time extraction. (Not tested)

in Transforms.conf

[stanza_name]
REPEAT_MATCH = true
REGEX = Elapsed\sProcessing\sTime\s::\s\[\w+\(([^\)]+)\),\s\w+\(([^\)]+)\),\s\w+\(([^\)]+)\)\].+?#\s\w+:\s(\d+)\s+\w+:\s[\w\.]+-([\w_]+)
FORMAT = $5-Max::$1 $5-Avg::$2 $5-Min::$3 $5-Total::$4

My regex may not be the most clean/efficient, but it appears to capture everything correctly on regexr.

So from:

Eventing Processing Time :: [Max(0.421), Avg(0.308), Min(0.194)] sec(s)      # Total: 00002      MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD

it will extract:

$1 = 0.421
$2 = 0.308
$3 = 0.194
$4 = 00002
$5 = RT_RTD_REVIEW_PERIOD

So $5-Max::$1 should become 'RT_RTD_REVIEW_PERIOD-Max = 0.421'

According to the documentation on transforms.conf, you can only do concatenated fields with index-time extractions.

mikelanghorst
Motivator

Hmm, only pulling the first line. REPEAT_MATCH = true is set, but no affect.

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...