How to use line breaking regex for Shibboleth audit logs?

[jwalzerpitt]

We are using Kafka Connect and we just started to ingest Shib audit logs. I am getting a bunch of events all rolled into one so I tried the splunk.hec.raw.line.breaker":"####", line in our JSON file on the Kafka Connect server as this matches the line I use on other log sources such as Cisco, Palo Alto, etc., and breaks the line just fine, but in this case, it's not working.

The events look like this:

{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}

What would be the regex to use to line break?

I tried using ,"EventTime":"(?>\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})" but the events aren't breaking quite right as I see:

-31 15:25:51"}{"EventReceivedTime":"2019-05-31

at the start of each event

Thx

[aromanauskas · Accepted solution]

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

[jwalzerpitt]

Worked perfectly - thx a million!