Splunk Search

How to use line breaking regex for Shibboleth audit logs?

jwalzerpitt
Influencer

We are using Kafka Connect and we just started to ingest Shib audit logs. I am getting a bunch of events all rolled into one so I tried the splunk.hec.raw.line.breaker":"####", line in our JSON file on the Kafka Connect server as this matches the line I use on other log sources such as Cisco, Palo Alto, etc., and breaks the line just fine, but in this case, it's not working.

The events look like this:

{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|username@foo.com|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}

What would be the regex to use to line break?

I tried using ,"EventTime":"(?>\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})" but the events aren't breaking quite right as I see:

-31 15:25:51"}{"EventReceivedTime":"2019-05-31

at the start of each event

Thx

0 Karma
1 Solution

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

View solution in original post

0 Karma

aromanauskas
Path Finder

This should get you what you're looking for. Since there is Nothing between your events you need an empty capture group. This will break if there are any closing braces in the events though.

In props.conf
LINE_BREAKER = }()
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = {"EventReceivedTime":"

If you are 100% sure that EventTime will always be at the end.
LINE_BREAKER = "EventTime":"\d+-\d+-\d+\s+\d+:\d+:\d+"}()

0 Karma

jwalzerpitt
Influencer

Worked perfectly - thx a million!

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...