All Apps and Add-ons

Will ignoreOlderThan permanently ignore a file or will a modtime change let the forwarder recheck the file?

Influencer

From the inputs.conf.spec ignoreOlderThan

Causes the monitored input to stop checking files for updates if their
modtime has passed this threshold.

We are monitoring Oracls DB audit trail files. The applications generates a separate file for each session. This can easily lead to thousands of files being created every hour. As such we need to set our ignoreOlderThan threshold very low (4h) to keep performance reasonable.

For 99% of these logs that is not a problem, but it's quite possible to have sessions that have a new entry appended after several hours. So the modtime will be updated. The last Answer I see on this topic (https://answers.splunk.com/answers/151149/does-splunk-re-index-a-file-that-was-ignored-due-t.html#co...) suggests that even though the modtime will change, if a file ever fell out of the ignoreOlderThan threshold it will NOT be checked unless the forwarder restarts.

Can anyone confirm if this is still the case in 6.3 + ?

1 Solution

Revered Legend

The behavior has not changed in 6.3/6.4.

View solution in original post

0 Karma

Ultra Champion

As a work-around, maybe you can increase the ignoreOlderThan by a day or so and exclude this day of data at the indexer level. We do pay, in such a case, for the license traffic for the extra day.

0 Karma

Influencer

For reference, we saw the forwarder memory usage spike at about 8GB when ignoreOlderThan was more than 4 hours.

Not Splunks fault, its just the way oracle writes its files out.

Also Batch mode is a non starter, as Oracle will not recreate audit session files after they are deleted.

0 Karma

Revered Legend

The behavior has not changed in 6.3/6.4.

View solution in original post

0 Karma

Influencer

Yeah that's what we saw from testing.

0 Karma

Community Manager
Community Manager

haha at first I only saw the title of your question and was about to share a post from 2 years ago related to this topic, but read through your entire explanation and saw you already referenced it *whistles and walks away...runs back* but I do hope you do get confirmation whether or not this forwarder behavior has changed 🙂 interesting topic!

Cheers!

0 Karma

Influencer

lol - plz my splunk answers fu is strong 🙂

0 Karma

Super Champion

any answer ?

0 Karma