topic Re: Multiple field extractions help in Splunk Search

Multiple field extractions help

attgjh1 — Wed, 06 Jun 2012 10:21:05 GMT

EDIT1: ive tweaked my regex abit. now i can extract the 'optional' fields i want. but im stumped at this particular log. ill post it here while i continuing to work on it as well 🙂

Summary:

{a}{b}{c}{d}{msg} are fields.

{a}{b}{msg} can be extracted

{a}{b}{c}{msg} can be extracted

{a}{b}{c}{d}{msg} can be extracted

having trouble with {msg} field that makes some 'message' unextractable. (i suspect this is the cause)

Details:

here are some sample logs:(extracted the specific portion)

my regex is as follows:

(?i),"?\[(?P<Instance>\w+)\] (?P<System>[^\.]+)\.(?P<Subsystem>[^\.\s]+)\.*(?P<Application>[^\.\s]*)?\.*(?P<Object>[^\s]*)?\s(?P<Message>.+)"?,(CLOSED|OPEN)

event 1:

,"[cng2] CIT.COS.MW (QUEUE.MANAGER)citos.cos.m msg pending, oldest 346s (exceed threshold) 04:15",CLOSED

Instance: cng2
System: CIT
Subsystem: COS
Application: MW
Object:
Message: (QUEUE.MANAGER)citos.cos.m msg pending, oldest 346s (exceed threshold) 04:15

event 2:

,"[OPS-TPTCC02] CIT.SS.COMMONS TException: [TS07] Disable sending function to TS, please contact administrators to resolve the issue.",OPEN,

Nothing extracted. However, there is actually another field after "Open" called severity that was extracted from this. So i found out about this missing event.

event 3:

,[cng1] CIT.Monitor Server is not being monitored,CLOSED,

Instance: cng1
System: CIT
Subsystem: Monitor
Application:
Object:
Message: Server is not being monitored

Re: Multiple field extractions help

kristian_kolb — Wed, 06 Jun 2012 10:47:03 GMT

Sorry, I can't really understand your regex, and what you expect to happen.

the leading comma would prohibit the Instance to match for any event. Also, I'm not sure that \w will match the dash in event 2.

What part of the events is System supposed to match? It's all lowercase, and no dots. The CITOS.* that comes after the [Instance] is uppercase = no match.

Please make a small matrix of what you want in each field for each event.

Re: Multiple field extractions help

attgjh1 — Thu, 07 Jun 2012 01:12:24 GMT

thanks for the tip. there was some odd \" in the logs that i overlooked. edited my regex and the optional fields worked!

but im stumped at log 2 atm. ill post it up anyway while i continue to work on it.

thanks a lot! 😄

Re: Multiple field extractions help

attgjh1 — Thu, 07 Jun 2012 01:37:42 GMT

Working Regex:

'(?i),"?\[(?P<Instance>.+)\] (?P<System>\w+)\.(?P<Subsystem>[^\.\s]+)\.*(?P<Application>[^\.\s]*)?\.*(?P<Object>[^\s]*)?\s(?P<Message>.+),(CLOSED|OPEN)'

Apparently i was using \w wrongly.

Re: Multiple field extractions help

kristian_kolb — Thu, 07 Jun 2012 06:18:49 GMT

That looks a lot better. Although I could suggest the following:
Instance: match on [^\]] instead of .+, or at least make it lazy; .+? , otherwise you might eat too much...

System: if the System can contain other characters than alphanumeric or underscore, i.e. \w, consider using the same pattern as for Subsystem or Application.