topic Re: Regex optional groups in Splunk Search

Regex optional groups

f1dot4 — Tue, 10 Mar 2015 17:46:02 GMT

Hi guys,
i'm trying to get this (simplified) regex running (for several days now):

^(?P<message>.+)(?:\s*SIP/2.0\s+(?P<sipaction>.+))?

i also tried another flavour of this statement:

^(?P<message>.+)(?:\s*SIP/2.0\s+(?P<sipaction>.+)|\w*)

With "Extract Fields" i tested this regex on splunkweb and it works, also when i click on "View in Search" it seems to be ok.

My problem is, that the field "sipaction" is not filled out with content and everything is written to "message" for new incoming indexed data. I tried to implement this regex for a new sourcetype via system/local/transforms.conf & props.conf and also via the Webinterface with the Field-Extractor tool.

I can break the problem down to the optional part of the statement:

^(?P<message>.+)[\s]*SIP/2.0\s+(?P<sipaction>.+)

In this case, "message" and "sipaction" is filled out, but i need the optional part (for a more complex regex).

There is also nothing special in var/log/splunk/.log*

My splunk version is 6.2.2-255606 on rhel 2.6.32-358.el6.x86_64, with splunk version 6.1.4, the optional-regex didn't work at all, and so i upgraded to 6.2.2 .

Some (simplified) data-examples from my log:

2015-03-10T18:20:08.647+01:00; INFO ; Sent final response to client: 
  SIP/2.0 480 Business Logic not available

2015-03-10T18:20:08.647+01:00; INFO ; Test

Thanks in advance!

Re: Regex optional groups

richgalloway — Tue, 10 Mar 2015 17:53:28 GMT

Your regex worked just fine on your sample data in regex101 with just a few minor changes.

 (?P<message>.+)(?:\s*SIP\/2\.0\s+(?P<sipaction>.+))?

Re: Regex optional groups

rsennett_splunk — Tue, 10 Mar 2015 17:56:59 GMT

it's a bit unclear as to exactly what you would like to extract. however... you are probably experiencing the result of having an unescaped delimeter. you have to escape the forward slash... check out your regex using regex101.com it's a great tool, and it will give you feedback so you can correct your stuff.

Re: Regex optional groups

f1dot4 — Wed, 11 Mar 2015 09:41:09 GMT

Thank you very much, i found a way to get my regex running, finally!

Only for documentation:
There seems to be a inconsistency between the "Extract Fields" regex checker on splunkweb and the regex-interpreter for incoming data. A short example:

(?:(?!User-Agent).)*(?:User-Agent:\s+(?P<useragent>[^\s]*))?

In this case

regex101 returns the useragent
the splunk regex checker on the "Extract Fields" page does NOT returns anything (?!)
when i implement this regex for a sourcetype, i get values for useragent-field at searchtime I always used the splunk regex-checker, regex101 seems to be a more reliable source! (splunk regex checker also works with my non-escaped character statement from above, while regex101 isn't)

Anyway, thanks for your help.

Example data:

2015-03-11T10:15:46.077+01:00; INFO ; HOSTNAME/P2928; 24; [RequestProcessor/ProcessBusinessLogicResponse]; Sent final response to client: SIP/2.0 480 Business Logic not available Via: SIP/2.0/UDP 0.0.0.0:0000;branch=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;rport=0000;received=0.0.0.0 To: ;tag=XXXXXXXX From: "" ;tag=XXXXXXXX Call-ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx CSeq: 2001 INVITE User-Agent: APP Allow: ACK, BYE, CANCEL, INFO, INVITE, NOTIFY, OPTIONS, REFER, REGISTER, SUBSCRIBE, UPDATE, MESSAGE Require: timer