Extracting multiple similar values from a multi-line event

mikelanghorst — Mon, 28 Sep 2020 13:31:06 GMT

I've got a rather tricky (at least for me) data set that I'd like to extract values from. For this example text

Elapsed Time Unti Event Locked Display :: Longest 5 Entries:
Elapsed Time Unti Event Locked Display :: [Max(0.361), Avg(0.180), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_RTD
Elapsed Time Unti Event Locked Display :: [Max(0.001), Avg(0.000), Min(0.000)] sec(s) # Total: 00007 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_WFC_STATUS
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00001 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_RTPD
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD
Elapsed Time Unti Event Locked Display :: [Max(0.000), Avg(0.000), Min(0.000)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_REVIEW_PERIOD
>>> Eventing Processing Time :: Longest 5 Entries:
Eventing Processing Time :: [Max(0.421), Avg(0.308), Min(0.194)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD
Eventing Processing Time :: [Max(0.198), Avg(0.194), Min(0.190)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_REVIEW_PERIOD
Eventing Processing Time :: [Max(0.149), Avg(0.142), Min(0.134)] sec(s) # Total: 00002 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_RTD
Eventing Processing Time :: [Max(0.039), Avg(0.039), Min(0.039)] sec(s) # Total: 00001 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTPD_RTPD
Eventing Processing Time :: [Max(0.017), Avg(0.011), Min(0.000)] sec(s) # Total: 00007 MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_WFC_STATUS
User Sessions :: Number of sessions 1
`

I need to extract out the Max,Avg,Min values for each MenuId: in Eventing Processing Time. Max isn't a multivalue field, but rather only relevent to the MenuId in the same line. So I'd need something like RT_RTD_REVIEW_PERIOD-Max and RT_RTD_REVIEW_PERIOD-Avg.

I'm not sure how to do this other than simple brute forcing with multiple regexes.

Re: Extracting multiple similar values from a multi-line event

emiller42 — Thu, 14 Mar 2013 17:36:09 GMT

I believe the following would work, but only as an index-time extraction. (Not tested)

in Transforms.conf

[stanza_name]
REPEAT_MATCH = true
REGEX = Elapsed\sProcessing\sTime\s::\s\[\w+\(([^\)]+)\),\s\w+\(([^\)]+)\),\s\w+\(([^\)]+)\)\].+?#\s\w+:\s(\d+)\s+\w+:\s[\w\.]+-([\w_]+)
FORMAT = $5-Max::$1 $5-Avg::$2 $5-Min::$3 $5-Total::$4

My regex may not be the most clean/efficient, but it appears to capture everything correctly on regexr.

So from:

Eventing Processing Time :: [Max(0.421), Avg(0.308), Min(0.194)] sec(s)      # Total: 00002      MenuId: rtdMenu.rtnIntervalDispatch.rtnIntervalDispExecuctionControl-RT_RTD_REVIEW_PERIOD

it will extract:

$1 = 0.421
$2 = 0.308
$3 = 0.194
$4 = 00002
$5 = RT_RTD_REVIEW_PERIOD

So $5-Max::$1 should become 'RT_RTD_REVIEW_PERIOD-Max = 0.421'

According to the documentation on transforms.conf, you can only do concatenated fields with index-time extractions.

Re: Extracting multiple similar values from a multi-line event

mikelanghorst — Fri, 15 Mar 2013 22:08:51 GMT

Hmm, only pulling the first line. REPEAT_MATCH = true is set, but no affect.

topic Extracting multiple similar values from a multi-line event in Splunk Search

Extracting multiple similar values from a multi-line event

Re: Extracting multiple similar values from a multi-line event

Re: Extracting multiple similar values from a multi-line event