topic Re: Need help with props on multiline event in Getting Data In

Need help with props on multiline event

banderson7 — Fri, 22 Jul 2016 15:09:06 GMT

We're bringing in syslog's from datapower units, and they have a rough log setup:

Jul 22 09:00:20 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214401)[request][100.73.230.24] gtid(214401):
Jul 22 09:00:20 10.214.8.104 Timestamp(2016-07-22T09:00:20-04:00)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            TransactionID()
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            ClientId(xxxxxx 4000025)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            UserInfo()
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            Status(0x00000000)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            TimeTake(V1_113_103_0_1_9)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            Message()

That's an event in syslog, but sometimes breaks in half in splunk:

Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                TimeTake(V1_207_187_0_0_20)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                Message()


Jul 21 23:15:44 172.22.93.21 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(33153921)[request][216.105.103.106]: 
Jul 21 23:15:44 172.22.93.21 Timestamp(2016-07-21T23:15:44-04:00)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                TransactionID()
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                ClientId(xxxxxx 4000025)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                UserInfo()
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                Status(0x00000000)

My props looks like this, and I'm pretty sure it's wrong, but not certain where:

[xxxxxx:prod:datapower]
#Dec 28 11:21:03 10.1.1.49 [cert-monitor][warn] cert-monitor(Certificate Monitor): trans(383): Certificate 'HummingbirdUATCert' is expired
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
#LINE_BREAKER = ([\n\r]+)(?=\w{3}\s+\d+\s+\d+:\d+:\d+)
LINE_BREAKER = ^(?:[^ \n]* ){5}(?P<reg>\[\d+\w+\])
TRUNCATE = 1024000
SHOULD_LINEMERGE = TRUE

Can someone help me stop these from coming in broken in half? I'm also interested in extracting the ClientId and Message fields, but w/ the amount of lines, am not getting far.

Re: Need help with props on multiline event

somesoni2 — Fri, 22 Jul 2016 15:22:57 GMT

Give this a try

[xxxxxx:prod:datapower]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=(\S+\s){4}\[)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 1024000

Re: Need help with props on multiline event

banderson7 — Fri, 22 Jul 2016 15:30:21 GMT

That brought in the entire syslog file as 1 entry.
Edit: and the entries that are coming in now are correct, and being divided at the right lines. Wonder why the file came in as 1 entry.

Re: Need help with props on multiline event

somesoni2 — Fri, 22 Jul 2016 15:38:20 GMT

Could you explain more on the issue that you see?

Re: Need help with props on multiline event

banderson7 — Tue, 29 Sep 2020 10:17:46 GMT

Sure. I ingested my syslog file w/ that props and it came in as one event:
Jul 22 09:00:20 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214401)[request][100.73.230.24] gtid(214401):
Jul 22 09:00:20 10.214.8.104 Timestamp(2016-07-22T09:00:20-04:00)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 TransactionID()
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 ClientId(xxxxxx 4000025)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 UserInfo()
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 Status(0x00000000)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 TimeTake(V1_113_103_0_1_9)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 Message()
Jul 22 09:00:55 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214449)[request][100.73.230.24] gtid(214449):
Jul 22 09:00:55 10.214.8.104 Timestamp(2016-07-22T09:00:55-04:00)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TransactionID()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 ClientId(xxxxxx 4000172)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 UserInfo()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Status(0x00000000)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TimeTake(V1_180_169_0_1_10)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Message()
Jul 22 09:01:56 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214513)[request][100.73.230.24] gtid(214513):
Jul 22 09:01:56 10.214.8.104 Timestamp(2016-07-22T09:01:56-04:00)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 TransactionID()
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 ClientId(xxxxxx 4000192)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 UserInfo()
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 Status(0x00000000)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 TimeTake(V1_151_138_0_0_13)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 Message()
with another 2000 lines. New events coming into splunk are broken up correctly, so the regex is correct. I'm uncertain how to break up the first event, and am uncertain if I want to try, depending on what's involved. Can you suggest any way to extract the ClientId and message fields?

Re: Need help with props on multiline event

somesoni2 — Fri, 22 Jul 2016 16:31:12 GMT

Doubt, so new events are working fine, then which ones are the one failing and where (splunk preview tool or actual data ingestion).?

Re: Need help with props on multiline event

somesoni2 — Fri, 22 Jul 2016 16:34:45 GMT

Try this for field extract
For Message Message\(\)([\r\n]*)(\S+\s+){4}(?<Message>.+)
For ClientId ClientId\((?<ClientId>[^\)]+)

Re: Need help with props on multiline event

mhassan — Fri, 22 Jul 2016 22:53:19 GMT

Can you explain where does the single event start and stop? What are the logical boundaries around an event? I see timestamp in every line (which in normal logs are single events) but they dont appears to be the case here.

Re: Need help with props on multiline event

banderson7 — Tue, 29 Sep 2020 10:21:41 GMT

This is a single event:
Jul 22 09:00:55 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214449)[request][100.73.230.24] gtid(214449):
Jul 22 09:00:55 10.214.8.104 Timestamp(2016-07-22T09:00:55-04:00)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TransactionID()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 ClientId(xxxxxx 4000172)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 UserInfo()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Status(0x00000000)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TimeTake(V1_180_169_0_1_10)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Message()