topic Re: Break Large Events into Many Small Events in Getting Data In

Break Large Events into Many Small Events

willial — Wed, 07 May 2014 15:33:35 GMT

I've run back and forth through the props.conf documentation and done a few circuits of Answers, but I haven't found anything that actually works yet, so here we are.

I have these large multi-line events coming into splunk with source=tcp-raw. They look kind of like this:

@@@
-> section 1
*bunch of stuff here*
@@@
-> section 2
*bunch of stuff here*
@@@

and so on. The "@@@" is what I want to break the events up on, since it's an obvious delimiter. So I tried adding the following to props.conf

[source::tcp-raw]
SHOULD_LINEMERGE = false
DATETIME_CONFIG = current
LINE_BREAKER = @@@

The result was splunk either not indexing the data at all, or throwing it out somehow. Whenever I'd send a new report with that stanza in place, nothing would show up. Removed it, and the data appeared in its normal unbroken form.

What am I doing wrong?

Re: Break Large Events into Many Small Events

somesoni2 — Mon, 28 Sep 2020 16:33:11 GMT

Try "BREAK_ONLY_BEFORE = @@@" instead of "LINE_BREAKER = @@@"

Re: Break Large Events into Many Small Events

willial — Wed, 07 May 2014 16:21:35 GMT

OK, that fixed the problem of splunk throwing out the incoming data, but it's still not breaking up the events.

Re: Break Large Events into Many Small Events

linu1988 — Wed, 07 May 2014 17:18:58 GMT

Did you re-index all the data or are you looking at the same data? It needs to be re-indexed.

Re: Break Large Events into Many Small Events

willial — Wed, 07 May 2014 17:40:18 GMT

Since I'm doing this in a test environment I'm able to throw a new batch of data at it on demand, which should then be indexed under the new rules. I'm following this process: change props.conf in /local, restart splunk server, send new data, check result, repeat.

Re: Break Large Events into Many Small Events

willial — Wed, 07 May 2014 18:05:47 GMT

I've solved my own problem. The stanza now looks like:

BREAK_ONLY_BEFORE = @@@
SHOULD_LINEMERGE = true

It's now splitting properly.