Field extraction for Log File Entries with Pipe delimiters

[mbasharat]

Hi,

I have a log file I am monitoring. Log file entries have pipe delimited field entries as below:

LE Variation 1:
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|... SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|

LE Variation 2:
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|...

Both variations exist in the log and I need both. The only differences among the two for distinction is that |success| defines successful transaction and anything other than |success| is a failure.

I need fields to be extracted using regex or eval in Splunk search please. You can rename them as samples and I will update at my end as needed.

Thanks in-advance.

[to4kawa · Accepted solution]

| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

[mbasharat]

Slight adjustment based on my use case and worked great. Thanks!!!

[mydog8it]

I think this will work for you...

Your_search | rex  "[|:](?P<Status>\D\w{6,8})[| ]" | stats count by Status

The caveat is the 'failure' messages would need to be non-digit and between 6&8 characters in length (otherwise adjust the regex). Careful modifying the length too much or it may introduce false positives.

[mbasharat]

Hi mydog8it,

First, I need to have the pipe delimited fields extracted in SPL search. For example:

task: [default task-11]
timestamp: 2020-01-23 12:45:01,851
loglevel: INFO
URL: sample.sample.sample.sample.sample.sample.StatLogger -
type: ABCStat
app: XYZ
id: 11111111111111111111
surl: http://www.abc.com/XYZ/123/ABCD/submission
timestamp2: 2020-01-24T13:10:52.414Z
timestamp3: 2020-01-24T13:10:54.595Z
transactionid: 2181
code: 0
submission: 3909
status: success

[richgalloway]

What have you tried so far?

[mbasharat]

I tried using field extractor but that will not do it because it should be done at the main config level. While that piece is being discussed, I need to go ahead and do search time extraction so I can create reports as needed.