Why is the multi-line event breaking working inconsistently?

responsys_cm — Tue, 29 Sep 2020 18:52:28 GMT

I'm trying to successfully ingest WebADM logs, a one-time password solution. The logs are... a mess. But the line breaking should be pretty straightforward, but the results are inconsistent.

This is what a successful login message looks like:

[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|New openotpSimpleLogin SOAP request|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Username: admin|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Domain: r1|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Password: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Client ID: LDAP|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Settings: ChallengeMode=No|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|> Options: -U2F|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Registered openotpSimpleLogin request|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Resolved LDAP user: cn=admin,ou=special,o=r1 (cached)|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Started transaction lock for user|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Found 37 user settings: LoginMode=LDAP,OTPType=TOKEN,OTPLength=6,ChallengeMode=Yes,ChallengeTimeout=90,ChallengeLock=No,ChallengeFake=No,TrustedContext=No,MobileTimeout=30,EnableLogin=Yes,TmpKeys=Yes,HOTPLookAheadWindow=25,TOTPTimeStep=30,TOTPTimeOffsetWindow=120,OCRASuite=OCRA-1:HOTP-SHA1-6:QN06-T1M,SMSType=Normal,SMSMode=Ondemand,MailMode=Ondemand,LastOTPTime=300,ListChallengeMode=ShowID,ReplyDataURL=http://webadm:4000/radius_authorization/authorize?client=10.10.5.20,http://webadm05:4000/radius_authorization/authorize?client=10.10.5.20,http://webadm04:4000/radius_authorization/authorize?client=10.10.5.20,http://webadm02:4000/radius_authorization/authorize?client=10.10.5.20,http://webadm-03301.node.ad3.r1:3000/radius_authorization/authorize?client=10.10.5.20,http://webadm03:4000/radius_authorization/authorize?client=10.10.5.20|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Found 1 request settings: ChallengeMode=No|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Found 2 user data: LoginCount,RejectCount|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Requested login factors: LDAP|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|LDAP password Ok|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Updated user data|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10
[2018-04-04 17:17:09] [10.10.10.10] CEF|0|RCDevs|WebADM|1.6.1-1|OpenOTP|Sent success response|1|sid=BKQSK2FC src=10.10.10.10 dst=10.10.10.10

For a failure message, the last line says "Sent failure response" instead.

I would think something like this should work:

[webadm]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = New openotpSimpleLogin SOAP request
MUST_BREAK_AFTER = Sent failure response|Sent success response

I've tried using only the "BREAK_ONLY_BEFORE" logic. I've tried including the TIME_FORMAT since I've read that can resolve some line breaking issues. I've tried replacing the spaces with "\s" regex.

I'd say the line breaking works about 85% correctly, but the 15% that don't work don't seem to have anything particular in common, such as the host, the WebADM server used, the user name, etc.

I feel like I'm taking crazy pills here. I cannot get Splunk to break these events consistently.

Re: Why is the multi-line event breaking working inconsistently?

Azeemering — Tue, 29 Sep 2020 18:56:09 GMT

What is your props setting now exactly?
What happens when you try this:

[webadm]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
MAX_TIMESTAMP_LOOKAHEAD=19
disabled=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
TIME_PREFIX=[

Also try if adding BREAK_ONLY_BEFORE=[\d{4}-\d\d?-\d\d?\s\d\d?:\d\d?:\d\d?] helps

topic Re: Why is the multi-line event breaking working inconsistently? in Getting Data In

Why is the multi-line event breaking working inconsistently?

Re: Why is the multi-line event breaking working inconsistently?