topic Re: Force splunk to poll files more quickly? in Getting Data In

Force splunk to poll files more quickly?

sconover — Mon, 11 Jul 2011 15:43:47 GMT

For testing purposes, I would really really like to force splunk to poll files in a monitor:// directory structure (and index changes) more quickly - at least temporarily. Is there some way I can force splunk to do this, overriding the default check-backoff behavior?

Re: Force splunk to poll files more quickly?

jbsplunk — Mon, 11 Jul 2011 16:12:31 GMT

There isn't a user configurable monitor interval setting which can be used to gain this kind of control over monitor stanzas. Generally speaking, I haven't seen the need for this though, as I see files indexed with almost no delay at all between when the files are updated and when the updates come into Splunk.

Re: Force splunk to poll files more quickly?

mikelanghorst — Mon, 11 Jul 2011 16:15:25 GMT

The only time I've seen this an issue with data taking too long to get there was with previous versions and trying to monitor a directory with too many files/subdirectories. Is your monitor stanza set overly broad then trying to whitelist down?

Re: Force splunk to poll files more quickly?

sconover — Mon, 11 Jul 2011 20:54:44 GMT

"Generally speaking, I haven't seen the need for this though, as I see files indexed with almost no delay at all between when the files are updated and when the updates come into Splunk."

My experiments show that there's anywhere from a zero to 15 second delay in indexing, which seems to match up well with the answer to this post:

http://splunk-base.splunk.com/answers/6693/how-to-force-splunk-to-index-new-files-quickly

I've since found a solution for what I want to happen - use oneshot:

http://splunk-base.splunk.com/answers/684/after-fixing-propsconf-how-to-re-index-the-same-files-using-the-new-settings

which converts indexing from async to blocking - more appropriate for my usage scenario.