Trouble splitting multi-line events

[muppetlegs]

I have some suricata stats logs which are in the following format:

------------------------------------------------------------------
Date: 9/26/2012 -- 16:57:53 (uptime: 0d, 00h 00m 24s)
------------------------------------------------------------------
Counter                   | TM Name                  | Value
------------------------------------------------------------------
tcp.sessions              | Detect                   | 2932
tcp.blah                  | Whatever                 | 42
...
string.string             | string                   | 0
------------------------------------------------------------------

I'm trying to :

1. extract four values: Counter, Module(for TM Name), Value, and Date
2. have the Counter, TM Name, and Value for each line within the event associate only with what's on its own line
3. except to take on the date value of the entire event

...but I'm not getting it done

My props.conf looks like:

[suricata_stats]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = \-*\r\nDate
EXTRACT-counter = ^(?<counter>[a-z]+\.[a-z]+)\s+|
EXTRACT-module = ^[a-z]+\.[a-z]+\s+|\s+(?<module>[^ ])\s+|
EXTRACT-value = ^[a-z]+\.[a-z]+\s+|\s+[^ ])\s+|\s+(?P<value>.+)

I know the regex is .. messy, but the results I'm getting are not as intended.

With sample data containing what i intend to be 5 results, I get 6.

One event is :
--------------------------------------------------------------------------

The next five events begin with "Date" and display all of the subsequent lines.

My extractions are also wrong:

1. the only value for "Counter" is "tcp.sessions" - no "tcp.blah", or others.
2. the only value for "Module" is "9" (from after "Date:")
3. i have five values for "Value", but each value is a multi-line value containing everything from "9/26/2012" down to "string.string"

Where do we start?

[dazole]

I'm working on the same thing, actually. I'm curious if you've managed to get a good setup for this. Here's what I have in my props.conf:

 [suricata_stats_log2]
BREAK_ONLY_BEFORE = Date:
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = False
TIME_PREFIX = Date:
pulldown_type = 1
EXTRACT-Counter = (?i)(?P<Counter>[^ ]+)\s+\|\s+\w+\s+\|\s+\d+
EXTRACT-TM_Name = (?i)\..*?   \| (?P<TM_Name>\w+)(?= )
EXTRACT-Value = (?i) .*?         \| (?P<Value>\d+)

Each line (tcp.sessions | Detect | 2932, etc) is it's own event with the correct timestamp. However, I'm not sure I like this particular solution yet. 272 actual stats turn into over 27000 events in Splunk! I'm still futzing with it to try to find something I like.

I've tried multikv and either it didn't work at all, or I was totally broken in my search implementation of it, heh.

[emiller42]

There is the multikv command to parse out tabular data at search-time, which would likely be easier than trying to do that in props.conf. But there is still the issue of getting the event in correctly in the first place, which I'm not sure about...