topic Re: Field extraction for Log File Entries with Pipe delimiters in Splunk Search

Field extraction for Log File Entries with Pipe delimiters

mbasharat — Fri, 14 Feb 2020 19:16:12 GMT

Hi,

I have a log file I am monitoring. Log file entries have pipe delimited field entries as below:

LE Variation 1:
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|

LE Variation 2:
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|

Both variations exist in the log and I need both. The only differences among the two for distinction is that |success| defines successful transaction and anything other than |success| is a failure.

I need fields to be extracted using regex or eval in Splunk search please. You can rename them as samples and I will update at my end as needed.

Thanks in-advance.

Re: Field extraction for Log File Entries with Pipe delimiters

richgalloway — Fri, 14 Feb 2020 19:38:57 GMT

What have you tried so far?

Re: Field extraction for Log File Entries with Pipe delimiters

mbasharat — Fri, 14 Feb 2020 19:49:46 GMT

I tried using field extractor but that will not do it because it should be done at the main config level. While that piece is being discussed, I need to go ahead and do search time extraction so I can create reports as needed.

Re: Field extraction for Log File Entries with Pipe delimiters

mydog8it — Fri, 14 Feb 2020 20:47:10 GMT

I think this will work for you...

Your_search | rex  "[|:](?P<Status>\D\w{6,8})[| ]" | stats count by Status

The caveat is the 'failure' messages would need to be non-digit and between 6&8 characters in length (otherwise adjust the regex). Careful modifying the length too much or it may introduce false positives.

Re: Field extraction for Log File Entries with Pipe delimiters

mbasharat — Fri, 14 Feb 2020 21:25:52 GMT

Hi mydog8it,

First, I need to have the pipe delimited fields extracted in SPL search. For example:

task: [default task-11]
timestamp: 2020-01-23 12:45:01,851
loglevel: INFO
URL: sample.sample.sample.sample.sample.sample.StatLogger -
type: ABCStat
app: XYZ
id: 11111111111111111111
surl: http://www.abc.com/XYZ/123/ABCD/submission
timestamp2: 2020-01-24T13:10:52.414Z
timestamp3: 2020-01-24T13:10:54.595Z
transactionid: 2181
code: 0
submission: 3909
status: success

Re: Field extraction for Log File Entries with Pipe delimiters

to4kawa — Sat, 15 Feb 2020 01:56:32 GMT

| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

Re: Field extraction for Log File Entries with Pipe delimiters

mbasharat — Tue, 25 Feb 2020 17:00:55 GMT

Slight adjustment based on my use case and worked great. Thanks!!!