topic Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B? in Splunk Search

How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

philallen1 — Fri, 29 Aug 2014 16:05:25 GMT

I have a problem in Splunk's regex and I can't figure it out for the life of me.

I'm going to simplify my problem a bit. Imagine this is my data:

|a|b|

If 'a' exists, I want my regex to pick out 'a' only, otherwise I want it to pick out 'b' only.

So in this case: |a|b| my regex should pick out 'a'. But, if for some reason 'a' didn't exist in my data, I want the same bit of regex to pick out 'b'.

|a|b| ---> should pick out 'a' only

|c|b| ---> should pick out 'b' only

'b' always exists, but 'a' sometimes doesn't.

Any ideas?

Thanks a lot!

Phil

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

ppablo — Fri, 29 Aug 2014 17:47:38 GMT

Hi @philallen1

This is a great question. It'll be really helpful if you could paste a sample of your data so people can help you figure out an accurate regex for your use case.

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

sk314 — Sat, 30 Aug 2014 12:02:01 GMT

Since "'b' always exists, but 'a' sometimes doesn't"
Have you tried using fillnull on field 'a'? like so:

[your search] | fillnull value=0 a | rex a "regex for field a" | rex b "regex for field b" | eval return_value = if(isnull(a), b, a)

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

lguinn2 — Sun, 31 Aug 2014 05:57:03 GMT

Why does it have to be "the same bit of regex"??

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

strive — Sun, 31 Aug 2014 17:45:46 GMT

You question clearly states that a and b are two separate fields, but your example shows that a is value of a field.

Assuming you will have a and b as two distinct fields, you need coalesce function.

try like this:
Your base search...| eval RequiredField = coalesce(a,b)

As @ppablo_splunk mentioned, if you can give sample log events, it will be helpful for us to answer your question.

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

philallen1 — Mon, 01 Sep 2014 09:04:05 GMT

Hi guys

Thanks for the responses. Here are some sample logs.

2014-08-29 16:09:00,830 INFO Order filled for 8=FIX.4.4|9=667|15=GBP|55=USD

2014-08-27 16:11:19,373 INFO Order filled for 8=FIX.4.4|9=667|55=USD

So the fields I am referring to are "15=..." and "55=...". I'm interested in the currency that each of these fields give.

The "15=..." field is the equivalent to my "a" field - i.e. it doesn't always exist. When it does exist I want the currency that it gives, otherwise I want the currency that "55=..." gives.

I hope that makes sense?

Phil

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

philallen1 — Mon, 01 Sep 2014 10:43:49 GMT

I'm not sure if I need to put these @ signs in to get people's attention, so I'm giving it a go!
@ppablo_splunk
@sk314
@lguinn
@strive

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

sk314 — Mon, 01 Sep 2014 14:03:24 GMT

try this: [your search query] | rex _raw "15=(?P<currency_a>\w+)" | rex _raw "55=(?P<currency_b>\w+)" | eval return_value = if( isnull(currency_a), currency_b, currency_a)

assuming the 'a' field is always preceded by '15=' and 'b' field is always preceded by '55='

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

Runals — Tue, 02 Sep 2014 10:07:11 GMT

I would try something like

... | rex "| \d+=(?[A-Z]+)"

In theory this will see the pipe followed by a number, equal sign, and then upper case letters and only do that for the first time it runs into that pattern. I forget if all currency has a 3 letter designate but you could potentially change the [A-Z]+ part to [A-Z]{3}.

Am going off the top of my head with no caffeine though so YMMV.

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

David — Thu, 11 Feb 2016 23:54:48 GMT

Probably better (definitely simpler) to do | eval return_value = coalesce(currency_a, currency_b)

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

sbotharaj — Fri, 30 Nov 2018 23:06:21 GMT

Worked perfectly for my situation

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

nick405060 — Fri, 17 Jan 2020 22:27:11 GMT

Here. _ fields are a little tricky so I would eval/rename them like I did here.

index=myindex | 

eval no_referrer_regex="MYREGEX1" |

eval referrer_regex="MYREGEX2" |

eval regex=if(_time < 1579250700,no_referrer_regex,referrer_regex) | eval raw=_raw |

map maxsearches=10000 search="| makeresults | eval mapped_raw=\"$$raw$$\" | rex field=mapped_raw \"$$regex$$\"" | table pst pst_epoch id action path num desc browser referrer

A second approach would just be to use ad-hoc searches in SimpleXML to set token values.

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

to4kawa — Sat, 18 Jan 2020 04:21:16 GMT

| makeresults 
| eval _raw="2014-08-29 16:09:00,830 INFO Order filled for 8=FIX.4.4|9=667|15=GBP|55=USD
2014-08-27 16:11:19,373 INFO Order filled for 8=FIX.4.4|9=667|55=USD" 
| makemv delim="
" _raw 
| stats count by _raw 
`comment("this is sample log")`
`comment("the logic")`
| rex "^(?<_time>[\w-]+ \d\d:\d\d:\d\d,\d{3}) .+ for (?<message>.+)" 
| eval _time=strptime(_time,"%F %T")
| eval message=split(message,"|")
| stats count by _time message
| rex field=message "(?<key>\w+)=(?<value>.+$)"
| eval {key} = value
| fields - message count key value
| stats values(*) as * by _time

This is @philallen1 's sample.
I think it is unnecessary to use conditional REGEX .

Re: How to write conditional regex to extract field A, but if field A does not exist, then extract field B?

woodcock — Sun, 19 Jan 2020 22:32:27 GMT

Try this (I may be taking you too literally, but the pieces that you need are here):

(?J)(?:^|[\r\n])\|(?:(?<MyField>a)\|b\|)|(?:c\|(?<MyField>b)\|)

https://regex101.com/r/5Peq78/1