Splunk Search

Field extraction for Log File Entries with Pipe delimiters

mbasharat
Builder

Hi,

I have a log file I am monitoring. Log file entries have pipe delimited field entries as below:

LE Variation 1:
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|... SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|

LE Variation 2:
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|...

Both variations exist in the log and I need both. The only differences among the two for distinction is that |success| defines successful transaction and anything other than |success| is a failure.

I need fields to be extracted using regex or eval in Splunk search please. You can rename them as samples and I will update at my end as needed.

Thanks in-advance.

0 Karma
1 Solution

to4kawa
Ultra Champion
| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

View solution in original post

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

0 Karma

mbasharat
Builder

Slight adjustment based on my use case and worked great. Thanks!!!

0 Karma

mydog8it
Builder

I think this will work for you...

Your_search | rex  "[|:](?P<Status>\D\w{6,8})[| ]" | stats count by Status

The caveat is the 'failure' messages would need to be non-digit and between 6&8 characters in length (otherwise adjust the regex). Careful modifying the length too much or it may introduce false positives.

0 Karma

mbasharat
Builder

Hi mydog8it,

First, I need to have the pipe delimited fields extracted in SPL search. For example:

task: [default task-11]
timestamp: 2020-01-23 12:45:01,851
loglevel: INFO
URL: sample.sample.sample.sample.sample.sample.StatLogger -
type: ABCStat
app: XYZ
id: 11111111111111111111
surl: http://www.abc.com/XYZ/123/ABCD/submission
timestamp2: 2020-01-24T13:10:52.414Z
timestamp3: 2020-01-24T13:10:54.595Z
transactionid: 2181
code: 0
submission: 3909
status: success

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What have you tried so far?

---
If this reply helps you, Karma would be appreciated.
0 Karma

mbasharat
Builder

I tried using field extractor but that will not do it because it should be done at the main config level. While that piece is being discussed, I need to go ahead and do search time extraction so I can create reports as needed.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...