Getting Data In

Why is Splunk line breaking a single IDS Alert event into two events?

dmenon
Explorer

Splunk is breaking ids single event into two events, such as:

4/11/16
2:42:46.152 PM 
04/11-14:42:46.152985 00:05:00:00:00:00 -> 00:00:00:05:00:01 type:0x800 len:0x222
10.20.30.40:59406 -> 106.120.151.145:80 TCP TTL:52 TOS:0x0 ID:53190 IpLen:20 DgmLen:532 DF
***A**** Seq: 0xBA0195C4  Ack: 0xBB15F92D  Win: 0x3E00  TcpLen: 20
[Xref => http://doc.emergingthreats.net/2008500]
host = ISMeta2 source = /var/log/snort/snort.log

4/11/16
2:42:46.000 PM 
[**] [1:2008500:6] ET MALWARE Sogoul.com Spyware User-Agent (SogouIMEMiniSetup) [**]
[Classification: A Network Trojan was Detected] [Priority: 1]
host = ISMeta2 source = /var/log/snort/snort.log

Which appears in snort.log as this one event:

[**] [1:2008500:6] ET MALWARE Sogoul.com Spyware User-Agent (SogouIMEMiniSetup) [**]
[Classification: A Network Trojan was Detected] [Priority: 1]
04/11-14:42:46.152985 00:05:00:00:00:00 -> 00:00:00:05:00:01 type:0x800 len:0x222
10.20.30.40:59406 -> 106.120.151.145:80 TCP TTL:52 TOS:0x0 ID:53190 IpLen:20 DgmLen:532 DF
***A**** Seq: 0xBA0195C4  Ack: 0xBB15F92D  Win: 0x3E00  TcpLen: 20
[Xref => http://doc.emergingthreats.net/2008500]

All events start with [**] I have props.conf configured as follows where snort_alert_full is sourcetype, but that doesn't fix my issue

[snort_alert_full]      
BREAK_ONLY_BEFORE = [**]

Thanks in advance

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

BREAK_ONLY_BEFORE expects a regular expression, [**] would be extremely malformed for your data. Have you tried \[\*\*\]?

That being said, you'd be better off utilizing LINE_BREAKER like this:

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\[\*\*\]
TIME_PREFIX = ^([^\r\n]+[\r\n]+){2}
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %m/%d-%H:%M:%S.%6N

That'll be MUCH faster and should achieve exactly what you need.

martin_mueller
SplunkTrust
SplunkTrust

These props.conf settings need to be where the parsing phase happens, that'll be a heavy forwarder or the indexers in this case. The universal forwarder will ignore these.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The LINE_BREAKER from my answer works fine using your sample event from the question. I've added timestamp parsing to the answer.

alt text

0 Karma

dmenon84
Path Finder

Hi, I think your answer will definitely work but I don't think I have the props.conf in the right place basically the logs are being forwarded to splunk through universal forwarder. Under etc/deployment-apps I see a folder SplunkUFSnortconfig folder which is where the inputs.conf is configured as to which location to monitor name of index,sourcetype etc. I created props.conf in SplunkUFSnortconfig -> local but that doesn't seem to do anything at all.

0 Karma

dmenon84
Path Finder

Some more sample events:

Event 1

[**] [1:2520169:2541] ET TOR Known Tor Exit Node UDP Traffic group 85 [**]
[Classification: Misc Attack] [Priority: 2]
04/12-11:22:01.201114 00:00:00:05:00:01 -> 00:05:00:00:00:00 type:0x800 len:0x53
93.158.215.174:23320 -> 90.80.70.60:53 UDP TTL:52 TOS:0x28 ID:43018 IpLen:20 DgmLen:69
Len: 41
[Xref => http://doc.emergingthreats.net/bin/view/Main/TorRules]

Event 2

[**] [1:2016104:3] ET TROJAN DNS Reply for unallocated address space - Potentially Malicious 1.1.1.0/24 [**]
[Classification: A Network Trojan was Detected] [Priority: 1]
04/12-11:30:31.770737 00:00:00:05:00:01 -> 00:05:00:00:00:00 type:0x800 len:0x69
125.39.136.74:53 -> 30.40.50.60:56038 UDP TTL:51 TOS:0x0 ID:35728 IpLen:20 DgmLen:91
Len: 63

and my current props.conf because I see [**} at 2 places so I expanded on your suggestion.

[snort_alert_full]
SHOULD_LINEMERGE = false
LINE_BREAKER =[**]\s+[\d+:\d+:\d+]
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That LINE_BREAKER cannot work. First, it doesn't have a capturing group for the linebreaker processor to consume and second, you're not escaping the regex special chars asterisk and square bracket.

0 Karma

dmenon84
Path Finder

Thanks for the help, this did not work, I think there is more to the way the logs are setup as well. For instance - I want splunk to look at event starting with [**] and then grab the timestamp from the third line of the log. How do I achieve that ?

[**] [1:2008500:6] ET MALWARE Sogoul.com Spyware User-Agent (SogouIMEMiniSetup) [**]
 [Classification: A Network Trojan was Detected] [Priority: 1]
 04/11-14:42:46.152985 00:05:00:00:00:00 -> 00:00:00:05:00:01 type:0x800 len:0x222
 10.20.30.40:59406 -> 106.120.151.145:80 TCP TTL:52 TOS:0x0 ID:53190 IpLen:20 DgmLen:532 DF
 ***A**** Seq: 0xBA0195C4  Ack: 0xBB15F92D  Win: 0x3E00  TcpLen: 20
 [Xref => http://doc.emergingthreats.net/2008500]

Thanks in advance.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...