topic Re: Regex to Select Data Between Line Breaks in Splunk Search

Regex to Select Data Between Line Breaks

dfrench151 — Thu, 11 Apr 2019 14:10:15 GMT

Hello,

I am trying to create a regex so that I can have all data in between line breaks as one event. Here is a sample of the data I'm working with:

isDraggingObject   : True
id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition          : {or, matches System.Object[] (?i)(^.*?host failure 
                     alert.*?www\.jennycraig\.com\.au.*?$), matches System.Object[] 
                     (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {route PVG22KK, severity warning}

isDraggingObject   : True
id                 : 3b5aa785-b854-4e43-900a-225da5786a27
condition          : {or, matches System.Object[] 
                     (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PVG22KK}

**isDraggingObject   : True
id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), 
                     matches System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PW0VV83}**

The aim is to get all data as one event.

Thanks in advance for you help. I've been trying multiple different regex expressions, but just can't figure it out...

Re: Regex to Select Data Between Line Breaks

richgalloway — Thu, 11 Apr 2019 14:52:58 GMT

Are you saying this data currently in multiple lines and you want to combine it into a single line? Do you want to do this at index time or search time?

Re: Regex to Select Data Between Line Breaks

dfrench151 — Wed, 30 Sep 2020 00:08:27 GMT

I want this to be completed at the time I index the date. Even 1 should be:
isDraggingObject : True
id : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition : {or, matches System.Object(^.?host failure
alert.?www.jennycraig.com.au.?$), matches System.Object(^.?\bwarning\b.?www.jennycraig.com.au.?$)}
catch_all : False
advanced_condition : {}
actions : {route PVG22KK, severity warning}

Event 2:
isDraggingObject : True
id : 3b5aa785-b854-4e43-900a-225da5786a27
condition : {or, matches System.Object(^.?\bcritical\b.?www.jennycraig.com.au.*?$)}
catch_all : False
advanced_condition : {}
actions : {severity critical, route PVG22KK}

Currently splunk is just grouping everything together into one event.

Re: Regex to Select Data Between Line Breaks

dmarling — Thu, 11 Apr 2019 18:37:01 GMT

Is this logging format some kind of application standard or is this something that someone made in house. I ask because the way it's currently formatted makes it difficult for index time field extractions. I have a search time extraction that can be used to accomplish what I believe you are trying to do (create key value pairs) using a run anywhere example with your data that was provided on your question:

| makeresults count=3 
| streamstats count as counter 
| eval _raw=case(counter=1, " isDraggingObject   : True
 id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
 condition          : {or, matches System.Object[] (?i)(^.*?host failure 
                      alert.*?www\.jennycraig\.com\.au.*?$), matches System.Object[] 
                      (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
 catch_all          : False
 advanced_condition : {}
 actions            : {route PVG22KK, severity warning}", counter=2, " isDraggingObject   : True
 id                 : 3b5aa785-b854-4e43-900a-225da5786a27
 condition          : {or, matches System.Object[] 
                      (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
 catch_all          : False
 advanced_condition : {}
 actions            : {severity critical, route PVG22KK}", counter=3, " **isDraggingObject   : True
 id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
 condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), 
                      matches System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
 catch_all          : False
 advanced_condition : {}
 actions            : {severity critical, route PW0VV83}**") 
| fields - counter _time 
| rex field=_raw mode=sed "s/(\n|^)\s+(\w+\s+):/█\1\2:/g" 
| makemv _raw delim="█" 
| rex mode=sed field=_raw "s/█//g"
| rex mode=sed field=_raw "s/\n/ /g"
| extract kvdelim=":" pairdelim="
"

The first sed statement is placing a unique character to be inserted into the event, which we can then use to make it a multi-valued results that splits each key value pairing into a unique value of the raw data. It then removes the unique character and then removes all line breaks and makes them spaces. After that is done you can run the extract command and it will produce proper key value pairing.

If this is a custom generated log event, I would suggest that they consider making it key value format by wrapping their values with quotes and changing the colon to an equal sign or follow the Windows events formatting standard if you want line breaks without quotes in your values.

Re: Regex to Select Data Between Line Breaks

robinettdonWY — Thu, 11 Apr 2019 20:20:39 GMT

Assuming you want everything in between the empty lines in 1 capture group (named "event")... this should work.

edit: sorry for cut and paste silliness.
(?<event>^.\S*[^\n<]*(?:(?:<(?!)|\n(?!$))[^\n]*)*(?:|\n$|\z))

Demo

Re: Regex to Select Data Between Line Breaks

dfrench151 — Wed, 30 Sep 2020 00:08:50 GMT

So would this be in the props.conf file?
LINE_BREAKER = (?^.\S*^\n<*(?:|\n$|\z))

Re: Regex to Select Data Between Line Breaks

robinettdonWY — Thu, 11 Apr 2019 21:08:02 GMT

look at my last edit.... i think you need to name the capture group (my example I named it "event" ).

https://regex101.com/r/aRu6NA/1

Re: Regex to Select Data Between Line Breaks

dfrench151 — Thu, 11 Apr 2019 21:56:41 GMT

The document is created from a powershell script that pulls data through an API connection with an online application. When the information is indexed, there are not line breaks...:

isDraggingObject   : True
id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition          : {or, matches System.Object[] (?i)(^.*?host failure alert.*?www\.jennycraig\.com\.au.*?$), matches 
                     System.Object[] (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {route PVG22KK, severity warning}
isDraggingObject   : True
id                 : 3b5aa785-b854-4e43-900a-225da5786a27
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PVG22KK}
isDraggingObject   : True
id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), matches 
                     System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PW0VV83}
isDraggingObject   : True
id                 : d9837013-68c9-42bf-a91f-8dd8a94ca377
condition          : {or, matches System.Object[] (?i)(^.*?host failure alert.*?www\.jennycraig\.com$), matches 
                     System.Object[] (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com$), matches System.Object[] 
                     (?i)(^.*?host failure alert.*?locations\.jennycraig\.com)...}
catch_all          : False
advanced_condition : {}
actions            : {route PW0VV83, severity warning}
isDraggingObject   : True
id                 : 20fc7d82-d17d-443e-9802-c8f2df462ce9
condition          : {or, equals System.Object[] critical: Page Failure alert on 
                     https://uw2pobi11.sonic.jennycraig.com:9503/analytics, equals System.Object[] critical: Page 
                     Failure alert on https://uw2pobi11.sonic.jennycraig.com:9501/console, equals System.Object[] 
                     critical: Page Failure alert on https://uw2pobi11.sonic.jennycraig.com:9501/em...}
catch_all          : False
advanced_condition : {}
actions            : {route PO77HX2}

I tried out your code in one of the searches and it produced no results.

Re: Regex to Select Data Between Line Breaks

dfrench151 — Thu, 11 Apr 2019 21:58:21 GMT

I tried that example as well and for some reason it is still combining all the data into one event.

Re: Regex to Select Data Between Line Breaks

woodcock — Fri, 12 Apr 2019 00:03:59 GMT

Your description is strange to me. If you mean that there is a blank line between events, then this will do it:

[<YourSourcetypeHere>]
SHOULD_LINEMERGE = false
LINEBREAKER = ([\r\n]\s*[\r\n]+)