Break Large Events into Many Small Events

[willial]

I've run back and forth through the props.conf documentation and done a few circuits of Answers, but I haven't found anything that actually works yet, so here we are.

I have these large multi-line events coming into splunk with source=tcp-raw. They look kind of like this:

@@@
-> section 1
*bunch of stuff here*
@@@
-> section 2
*bunch of stuff here*
@@@

and so on. The "@@@" is what I want to break the events up on, since it's an obvious delimiter. So I tried adding the following to props.conf

[source::tcp-raw]
SHOULD_LINEMERGE = false
DATETIME_CONFIG = current
LINE_BREAKER = @@@

The result was splunk either not indexing the data at all, or throwing it out somehow. Whenever I'd send a new report with that stanza in place, nothing would show up. Removed it, and the data appeared in its normal unbroken form.

What am I doing wrong?

[willial]

I've solved my own problem. The stanza now looks like:

BREAK_ONLY_BEFORE = @@@
SHOULD_LINEMERGE = true

It's now splitting properly.

[willial]

Since I'm doing this in a test environment I'm able to throw a new batch of data at it on demand, which should then be indexed under the new rules. I'm following this process: change props.conf in /local, restart splunk server, send new data, check result, repeat.

[linu1988]

Did you re-index all the data or are you looking at the same data? It needs to be re-indexed.

[willial]

OK, that fixed the problem of splunk throwing out the incoming data, but it's still not breaking up the events.

[somesoni2]

Try "BREAK_ONLY_BEFORE = @@@" instead of "LINE_BREAKER = @@@"