Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

Need help with props on multiline event

banderson7
Communicator
‎07-22-2016 08:09 AM

We're bringing in syslog's from datapower units, and they have a rough log setup:

Jul 22 09:00:20 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214401)[request][100.73.230.24] gtid(214401):
Jul 22 09:00:20 10.214.8.104 Timestamp(2016-07-22T09:00:20-04:00)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            TransactionID()
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            ClientId(xxxxxx 4000025)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            UserInfo()
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            Status(0x00000000)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            TimeTake(V1_113_103_0_1_9)
Jul 22 09:00:20 10.214.8.104                            ::
Jul 22 09:00:20 10.214.8.104                            Message()

That's an event in syslog, but sometimes breaks in half in splunk:

Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                TimeTake(V1_207_187_0_0_20)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                Message()


Jul 21 23:15:44 172.22.93.21 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(33153921)[request][216.105.103.106]: 
Jul 21 23:15:44 172.22.93.21 Timestamp(2016-07-21T23:15:44-04:00)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                TransactionID()
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                ClientId(xxxxxx 4000025)
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                UserInfo()
Jul 21 23:15:44 172.22.93.21                ::
Jul 21 23:15:44 172.22.93.21                Status(0x00000000)

My props looks like this, and I'm pretty sure it's wrong, but not certain where:

[xxxxxx:prod:datapower]
#Dec 28 11:21:03 10.1.1.49 [cert-monitor][warn] cert-monitor(Certificate Monitor): trans(383): Certificate 'HummingbirdUATCert' is expired
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
#LINE_BREAKER = ([\n\r]+)(?=\w{3}\s+\d+\s+\d+:\d+:\d+)
LINE_BREAKER = ^(?:[^ \n]* ){5}(?P<reg>\[\d+\w+\])
TRUNCATE = 1024000
SHOULD_LINEMERGE = TRUE

Can someone help me stop these from coming in broken in half? I'm also interested in extracting the ClientId and Message fields, but w/ the amount of lines, am not getting far.

0 Karma
Reply
1 Solution
Solved! Jump to solution
Solution

somesoni2
Revered Legend
‎07-22-2016 08:22 AM

Give this a try

[xxxxxx:prod:datapower]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=(\S+\s){4}\[)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 1024000

View solution in original post

0 Karma
Reply
Solved! Jump to solution

mhassan
Path Finder
‎07-22-2016 03:53 PM

Can you explain where does the single event start and stop? What are the logical boundaries around an event? I see timestamp in every line (which in normal logs are single events) but they dont appears to be the case here.

0 Karma
Reply
Solved! Jump to solution

banderson7
Communicator
‎07-25-2016 06:22 AM

This is a single event:
Jul 22 09:00:55 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214449)[request][100.73.230.24] gtid(214449):
Jul 22 09:00:55 10.214.8.104 Timestamp(2016-07-22T09:00:55-04:00)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TransactionID()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 ClientId(xxxxxx 4000172)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 UserInfo()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Status(0x00000000)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TimeTake(V1_180_169_0_1_10)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Message()

0 Karma
Reply
Solved! Jump to solution
Solution

somesoni2
Revered Legend
‎07-22-2016 08:22 AM

Give this a try

[xxxxxx:prod:datapower]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=(\S+\s){4}\[)
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 1024000
0 Karma
Reply
Solved! Jump to solution

banderson7
Communicator
‎07-22-2016 08:30 AM

That brought in the entire syslog file as 1 entry.
Edit: and the entries that are coming in now are correct, and being divided at the right lines. Wonder why the file came in as 1 entry.

0 Karma
Reply
Solved! Jump to solution

somesoni2
Revered Legend
‎07-22-2016 08:38 AM

Could you explain more on the issue that you see?

0 Karma
Reply
Solved! Jump to solution

banderson7
Communicator
‎07-22-2016 08:43 AM

Sure. I ingested my syslog file w/ that props and it came in as one event:
Jul 22 09:00:20 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214401)[request][100.73.230.24] gtid(214401):
Jul 22 09:00:20 10.214.8.104 Timestamp(2016-07-22T09:00:20-04:00)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 TransactionID()
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 ClientId(xxxxxx 4000025)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 UserInfo()
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 Status(0x00000000)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 TimeTake(V1_113_103_0_1_9)
Jul 22 09:00:20 10.214.8.104 ::
Jul 22 09:00:20 10.214.8.104 Message()
Jul 22 09:00:55 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214449)[request][100.73.230.24] gtid(214449):
Jul 22 09:00:55 10.214.8.104 Timestamp(2016-07-22T09:00:55-04:00)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TransactionID()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 ClientId(xxxxxx 4000172)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 UserInfo()
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Status(0x00000000)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 TimeTake(V1_180_169_0_1_10)
Jul 22 09:00:55 10.214.8.104 ::
Jul 22 09:00:55 10.214.8.104 Message()
Jul 22 09:01:56 10.214.8.104 [0x80c0003f][xxxxxxSSOSPDebug][info] xmlfirewall(SSOAuditLogFW): trans(214513)[request][100.73.230.24] gtid(214513):
Jul 22 09:01:56 10.214.8.104 Timestamp(2016-07-22T09:01:56-04:00)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 TransactionID()
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 ClientId(xxxxxx 4000192)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 UserInfo()
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 Status(0x00000000)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 TimeTake(V1_151_138_0_0_13)
Jul 22 09:01:56 10.214.8.104 ::
Jul 22 09:01:56 10.214.8.104 Message()
with another 2000 lines. New events coming into splunk are broken up correctly, so the regex is correct. I'm uncertain how to break up the first event, and am uncertain if I want to try, depending on what's involved. Can you suggest any way to extract the ClientId and message fields?

0 Karma
Reply
Solved! Jump to solution

somesoni2
Revered Legend
‎07-22-2016 09:34 AM

Try this for field extract
For Message Message\(\)([\r\n]*)(\S+\s+){4}(?<Message>.+)
For ClientId ClientId\((?<ClientId>[^\)]+)

0 Karma
Reply
Solved! Jump to solution

somesoni2
Revered Legend
‎07-22-2016 09:31 AM

Doubt, so new events are working fine, then which ones are the one failing and where (splunk preview tool or actual data ingestion).?

0 Karma
Reply
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...