Why is time_before_close attribute causing a delay in indexing ?

[nibinabr]

I had set the value of time_before_close attribute to 300 (5 mins) in one of my monitor stanzas. What I observed is that splunk indexes the logs for the first few seconds and then stops indexing for the next 5 mins. After 5 mins next set of logs are indexed by splunk and then continues to wait for 5 mins.

I was expecting that time_before_close attribute would just keep the file handler open for the specified seconds after the last line was written in to the file but with continues indexing.

Any thoughts on why I see this behavior ? Or can someone direct me to somewhere I can find the answer ?

[doitslu]

I know this question is older but I will try to answer it since I hit on it an think maybe the answer is interesting for the 765 guys that have seen this question till now and the ones that will see this question.

By default Splunk will only open up to 100 file descriptors for reading logs (limits.conf : max_fd).
So if you define time_before_close and monitor more than the limit of file descriptors splunk first has to wait until a file discriptor becomes free before it can start reading the next file.

So if you have many monitored files splunk will imediatly start reading the first 100 files, wait time_before_close (in your case 5mins) and after that starts reading the next 100 files ...