Props.conf for log entry with multiple formats

[arunsoni]

The below log entry includes different format within it. Not sure how to write props.conf for proper field extractions and line breaking. each log entry has text, delimitter(|) and json.

2024-03-11T20:58:12.605Z [INFO] SessionManager sgrp:System_default swn:99999 sreq:1234567 | {"abrMode":"NA","abrProto":"HLS","event":"Create","sUrlMap":"","sc":{"Host":"x.x.x.x","OriginMedia":"HLS","URL":"/x.x.x.x/vod/Test-XXXX/XXXXX.smil/transmux/XXXXX"},"sm":{"ActiveReqs":0,"ActiveSecs":0,"AliveSecs":360,"MediaSecs":0,"SpanReqs":0,"SpanSecs":0},"swnId":"XXXXXXXX","wflow":"System_default"}
2024-03-11T20:58:12.611Z [INFO] SessionManager sgrp:System_default swn:99999 sreq:1234567 | {"abrMode":"NA","abrProto":"HLS","event":"Cache","sUrlMap":"","sc":{"Host":"x.x.x.x","OriginMedia":"HLS","URL":"/x.x.x.x/vod/Test-XXXXXX/XXXXXX.smil/transmux/XXX"},"sm":{"ActiveReqs":0,"ActiveSecs":0,"AliveSecs":0,"MediaSecs":0,"SpanReqs":0,"SpanSecs":0},"swnId":"XXXXXXXXXXXXX","wflow":"System_default"}

[KendallW]

Hi @arunsoni,

For this data you can break the events before each timestamp appears.
Splunk probably won't be able to handle the JSON field extraction because of the preamble line, so just extract the fields at search time.

props.conf :

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 25

search-time field extraction regex:

(?<abrMode>[^"]+)"abrMode":"(?<abrMode>[^"]+)","abrProto":"(?<abrProto>[^"]+)","event":"(?<event>[^"]+)","sUrlMap":"(?<sUrlMap>[^"]+)","sc":\{"Host":"(?<Host>[^"]+)","OriginMedia":"(?<OriginMedia>[^"]+)","URL":"(?<URL>[^"]+)"\},"sm":\{"ActiveReqs":(?<ActiveReqs>\d+),"ActiveSecs":(?<ActiveSecs>\d+),"AliveSecs":(?<AliveSecs>\d+),"MediaSecs":(?<MediaSecs>\d+),"SpanReqs":(?<SpanReqs>\d+),"SpanSecs":(?<SpanSecs>\d+)\},"swnId":"(?<swnId>[^"]+)","wflow":"(?<wflow>[^"]+)"

[arunsoni]

Search level extraction : I am thinking instead of json extraction I think it could be easy to add index extraction as json so that each event will treat as json message and the remaining few lines we can do extraction at the search level. let me know your thoughts ? @KendallW 

[arunsoni]

I want to do everything at the indexer level. Any idea on how to handle this data set ?