topic Re: How to configure Splunk to not index a line before it is finished writing? in Getting Data In

How to configure Splunk to not index a line before it is finished writing?

BrendanMcE — Fri, 21 Nov 2014 16:43:42 GMT

We are writing out to a log for which splunk is indexing for most lines okay, but some times splunk indexes before the line has finished writing.

This is due to the process in the way the log line is generated. Is there a way to tell splunk to not index the line until the next line is seen?

Re: How to configure Splunk to not index a line before it is finished writing?

cpetterborg — Fri, 21 Nov 2014 19:55:50 GMT

If you are indexing a log file that is tied to a process that is sending buffered output, then you will always have a problem (from my experience). I had a couple of processes that did that, and I had to force the output of the complete buffer, even if it was only partly full. One example was a Curl program that collected output and put it into a file that Splunk indexed. The curl invocation had to be done with the flag that told it not to buffer the output. If I didn't do this, then it would split lines all over the place as it wrote out 4096 bytes at a time.

Is this the type of thing you are seeing?

Re: How to configure Splunk to not index a line before it is finished writing?

BrendanMcE — Mon, 24 Nov 2014 10:38:51 GMT

We are using Jmeter and it starts writing out the line, then adds some more and so on until the line is complete. Splunk indexes it partially.
Just thought, there might be a way to stop splunk indexing the line until it sees the start of the next line, say the date.

Re: How to configure Splunk to not index a line before it is finished writing?

felipetesta — Mon, 28 Sep 2020 18:14:12 GMT

How is the Jmeter input configured? Are you sure that there is no "backslash_r" or "backslash_n" hidden in the slow log line? See http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Indexmulti-lineevents for info on event breaking.

Re: How to configure Splunk to not index a line before it is finished writing?

cpetterborg — Mon, 28 Sep 2020 18:15:07 GMT

Have you tried MUST_NOT_BREAK_BEFORE set to a newline or carriage return or both?

Re: How to configure Splunk to not index a line before it is finished writing?

shbagautdinov — Tue, 29 Sep 2020 12:32:43 GMT

Hi,
I know that this is the old question, but it would have saved my day if it was answered before )

The answer is to add time_before_close=60 (or another integer) into inputs.conf and all events will index correctly!

https://answers.splunk.com/answers/103132/events-are-broken-in-the-middle-of-the-line.html
https://answers.splunk.com/answers/492950/the-app-is-indexing-event-before-the-tmg-has-write.html