Splunk Search

How to use line breaking regex for Shibboleth audit logs?

jwalzerpitt
Influencer

We are using Kafka Connect and we just started to ingest Shib audit logs. I am getting a bunch of events all rolled into one so I tried the splunk.hec.raw.line.breaker":"####", line in our JSON file on the Kafka Connect server as this matches the line I use on other log sources such as Cisco, Palo Alto, etc., and breaks the line just fine, but in this case, it's not working.

The events look like this:

{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}

What would be the regex to use to line break?

I tried using ,"EventTime":"(?>\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})" but the events aren't breaking quite right as I see:

-31 15:25:51"}{"EventReceivedTime":"2019-05-31

at the start of each event

Thx

0 Karma
1 Solution

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

View solution in original post

0 Karma

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

0 Karma

jwalzerpitt
Influencer

Worked perfectly - thx a million!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...