topic Re: LINEBREAKER Help with snort logs in Getting Data In

LINEBREAKER Help with snort logs

I-Man — Tue, 20 Sep 2011 17:39:54 GMT

All,

Below are the logs prior to splunk interpreting them. I want to split each event with a regex based on the lines of =+=+=+=+=+=+...which is consistently 37 patterns long.

[**] [1:2008581:3] ET P2P BitTorrent DHT ping request [**]
[Classification: Potential Corporate Privacy Violation] [Priority: 1]
[Xref => http://www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/P2P/P2P_Bittorrent_Traffic]
[Xref => http://doc.emergingthreats.net/bin/view/Main/2008581]
[Xref => http://wiki.theory.org/BitTorrentDraftDHTProtocol]
Event ID: 2     Event Reference: 2
09/19/11-23:09:15.740430 10.x.x.x:4566 -> x.x.x.x:59965
UDP TTL:126 TOS:0x0 ID:31114 IpLen:20 DgmLen:95
Len: 75
64 31 3A 61 64 32 3A 69 64 32 30 3A 49 78 E1 96  d1:ad2:id20:Ix..
61 63 FA 84 13 3A 96 C1 F6 76 DC 53 DE 87 CC 95  ac...:...v.S....
65 31 3A 71 34 3A 70 69 6E 67 31 3A 74 34 3A 67  e1:q4:ping1:t4:g
3B 36 79 31 3A 76 34 3A 55 54 62 B9 31 3A 79 31  ;6y1:v4:UTb.1:y1
3A 71 65                                         :qe

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

[**] [1:2010140:5] ET P2P Vuze BT UDP Connection [**]
[Classification: Potential Corporate Privacy Violation] [Priority: 1]
[Xref => http://www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/P2P/P2P_Vuze]
[Xref => http://doc.emergingthreats.net/2010140]
[Xref => http://vuze.com]
Event ID: 3     Event Reference: 3
09/19/11-23:09:15.889491 10.x.x.x:60653 -> x.x.x.x:35421
UDP TTL:62 TOS:0x0 ID:51471 IpLen:20 DgmLen:93
Len: 73
9E 66 87 A7 D0 2A 93 5F 00 00 04 06 96 3E 45 1A  .f...*._.....>E.
32 00 00 00 00 00 32 04 CC 45 B6 01 EC ED E5 CE  2.....2..E......
09 70 00 00 01 32 83 F3 59 3E 14 41 7E D1 0C 90  .p...2..Y>.A~...
4D 4C 51 5D 69 43 F5 4C F1 46 00 BB EE C3 C2 02  MLQ]iC.L.F......
1E                                               .

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

Thanks,
Iman

UPDATE
With the following config, here are some results splunk is giving us:

[host::sensor01]
BREAK_ONLY_BEFORE_DATE = false
LINE_BREAKER = ([\r\n]*[=+]{74}[\r\n]*)

Results:

09/20/11-16:16:07.511888 10.255.4.201:52954 -> 64.208.138.115:80
TCP TTL:63 TOS:0x0 ID:2122 IpLen:20 DgmLen:48
******S* Seq: 0x4B5C56AC  Ack: 0x0  Win: 0xFFFF  TcpLen: 28
TCP Options (3) => MSS: 1460 SackOK EOL 
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
[**] [1:2408054:271] ET RBN Known Malvertiser IP TCP (28) [**]
[Classification: Misc Attack] [Priority: 2]
[Xref => http://doc.emergingthreats.net/bin/view/Main/RussianBusinessNetwork]
Event ID: 55744     Event Reference: 55744

[**] [1:2010140:5] ET P2P Vuze BT UDP Connection [**]
[Classification: Potential Corporate Privacy Violation] [Priority: 1]
[Xref => http://www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/P2P/P2P_Vuze]
[Xref => http://doc.emergingthreats.net/2010140]
[Xref => http://vuze.com]
Event ID: 6519     Event Reference: 6519
09/20/11-20:16:05.267902 10.250.85.60:6881 -> 2.30.187.43:4860
UDP TTL:62 TOS:0x0 ID:60117 IpLen:20 DgmLen:93
Len: 73
E7 03 B0 4C 87 80 D6 64 00 00 04 06 D8 6C BE 4D  ...L...d.....l.M

No consistency with the returned logs, not sure what I'm missing.

Re: LINEBREAKER Help with snort logs

kristian_kolb — Tue, 20 Sep 2011 19:32:25 GMT

Not really too good with regexes, but shouldn't it say

LINE_BREAKER = ([\r\n]+[=+]{74}[\r\n]+)

or does this amount to the same?

Added the extra newlines so that they are also removed from the event.
You might also want to increase the MAX_TIMESTAMP_LOOKAHEAD to value large enough for Splunk to detect it. The default value is 150, and it looks like your timestamps are further into the event than that.

Re: LINEBREAKER Help with snort logs

Ayn — Tue, 20 Sep 2011 19:57:21 GMT

Did you have a look at the Splunk for Snort app? It's been a while since I looked at raw Snort logs but I think your logs are not in the default alert-full format, but the app comes with props.conf settings that should get you going.

Re: LINEBREAKER Help with snort logs

I-Man — Tue, 20 Sep 2011 20:22:11 GMT

Tried this and edited my original Post. No luck... thanks anyways.

Re: LINEBREAKER Help with snort logs

Ayn — Mon, 28 Sep 2020 09:53:30 GMT

My guess is that the problem lies in that you've set a LINE_BREAKER but haven't told Splunk not to perform line merging. From the docs:

LINE_BREAKER = <regular expression>
* Specifies a regex that determines how the raw text stream is broken into initial events, before line merging takes place. (See the SHOULD_LINEMERGE attribute, below)
[...]
SHOULD_LINEMERGE = [true|false]
* When set to true, Splunk combines several lines of data into a single multiline event, based on the following configuration attributes.
* Defaults to true.

So, while the events may be broken up correctly by the custom LINE_BREAKER (I haven't tried the regex myself), Splunk will merge them all together again anyway and break after 256 lines (according to the MAX_EVENTS directive).

You could either disable line merging by setting SHOULD_LINEMERGE to false, or you could use a default line breaker and break your events by using line merging rules instead. The ones I use in the Splunk for Snort app are as follows:

SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = \[\*\*\] \[\d+\:\d+\:\d+\]
TIME_PREFIX     = (?=\d+/\d+)
TIME_FORMAT     = %m/%d-%T.%6N