Getting Data In

How does Splunk handle end of file?

eregon
Path Finder

Good evening fellow Splunkthiasts, can anyone explain in detail, how Splunk breaks the events when it finds the end of file with no EOL (or whatever LINE_BREAKER is set to)? Specifically I am concerned whether and how it recognizes the difference between "there is nothing more to read, the last event can be indexed as is" and "the last event has not been completely written yet, indexing should wait a bit more".

After some attempts and failures it seems to me that each event data is being read into some kind of buffer and when EOF with no EOL is reached, Splunk waits up to 10 seconds for more data coming in. If there is no further input within this time-limit, event is considered to be complete as it is stored in the buffer and is pushed to further processing.

Example: this Bash one-liner writes a single line to the file, but Splunk ingests two events, although there is no newline character:

(echo -n $(date '+%Y-%m-%d %H:%M:%S' "This is a log record from very sssssllloooo"; sleep 11s; echo "oooooow source.") > monitoredFile.log

Note: I am getting the same behavior when I redirect the one-liner's output to TCP input instead of monitored file.

I have two questions then:

  1. Is my understanding of the situation correct?
  2. If it is, can the 10s delay be adjusted? How? Can it be done for some specific sourcetype or host only?

Thanks in advance!

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
  1. I believe you are correct. Your testing would seem confirm my expectations.
  2. Check out the time_before_close and multiline_event_extra_waittime settings in inputs.conf.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust
  1. I believe you are correct. Your testing would seem confirm my expectations.
  2. Check out the time_before_close and multiline_event_extra_waittime settings in inputs.conf.
---
If this reply helps you, Karma would be appreciated.

eregon
Path Finder

Perfect, @richgalloway ! My mistake was to expect these settings in props.conf. I did not realize inputs.conf is much more logical place for such configuration, that's why I did not find it myself. Thanks for your help!

0 Karma
Get Updates on the Splunk Community!

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...