https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454764#M78708 <P>Perfect, @richgalloway ! My mistake was to expect these settings in props.conf. I did not realize inputs.conf is much more logical place for such configuration, that's why I did not find it myself. Thanks for your help!</P> Wed, 21 Aug 2019 09:09:40 GMT eregon 2019-08-21T09:09:40Z https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454762#M78706 <P>Good evening fellow Splunkthiasts, can anyone explain in detail, how Splunk breaks the events when it finds the end of file with no EOL (or whatever LINE_BREAKER is set to)? Specifically I am concerned whether and how it recognizes the difference between "there is nothing more to read, the last event can be indexed as is" and "the last event has not been completely written yet, indexing should wait a bit more".</P> <P>After some attempts and failures it seems to me that each event data is being read into some kind of buffer and when EOF with no EOL is reached, Splunk waits up to 10 seconds for more data coming in. If there is no further input within this time-limit, event is considered to be complete as it is stored in the buffer and is pushed to further processing.</P> <P>Example: this Bash one-liner writes a single line to the file, but Splunk ingests two events, although there is no newline character:</P> <P><CODE>(echo -n $(date '+%Y-%m-%d %H:%M:%S' "This is a log record from very sssssllloooo"; sleep 11s; echo "oooooow source.") &gt; monitoredFile.log</CODE></P> <P>Note: I am getting the same behavior when I redirect the one-liner's output to TCP input instead of monitored file.</P> <P><STRONG>I have two questions then:</STRONG></P> <OL> <LI>Is my understanding of the situation correct?</LI> <LI>If it is, can the 10s delay be adjusted? How? Can it be done for some specific sourcetype or host only?</LI> </OL> <P>Thanks in advance!</P> Tue, 20 Aug 2019 16:21:12 GMT https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454762#M78706 eregon 2019-08-20T16:21:12Z https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454763#M78707 <OL> <LI>I believe you are correct. Your testing would seem confirm my expectations.</LI> <LI>Check out the <CODE>time_before_close</CODE> and <CODE>multiline_event_extra_waittime</CODE> settings in inputs.conf.</LI> </OL> Tue, 20 Aug 2019 19:38:17 GMT https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454763#M78707 richgalloway 2019-08-20T19:38:17Z https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454764#M78708 <P>Perfect, @richgalloway ! My mistake was to expect these settings in props.conf. I did not realize inputs.conf is much more logical place for such configuration, that's why I did not find it myself. Thanks for your help!</P> Wed, 21 Aug 2019 09:09:40 GMT https://community.splunk.com/t5/Getting-Data-In/How-does-Splunk-handle-end-of-file/m-p/454764#M78708 eregon 2019-08-21T09:09:40Z