Getting Data In

Issues of 'ignoreOlderThan' in inputs.conf setting

jichen
Explorer

Hi,guys,in my scenario,a universal forwarder(UF vertion 4.3.2 for aix) monitores about 700 small files, the cpu usage will be more than 60%. I set a parameter 'ignoreOlderThan=1d' to reduce the cpu usage,but once the modtime of these files changed I want these updated data collected. After this configuration,the data updated in these small files after 23:59:59 till next day will not be collected by splunk agent. So I wanna ask below questions:
1.The mechanism of 'ignoreOlderThan',not up to the file's modtime,but the continuous time that splunk forwarder monitored it since the agent start? Otherwise,why the updated data will be ignored?
2.I did some test,I created a file at 10:00am ,and I set 'ignoreOlderThan=60s' in inputs.conf to monitor it, then I start the agent at 10:05am, this file was still indexed.
I know in the instruction document,it sayes 'ignoreOldThan' checking file by the modtime, but in my case it doesn't work. I haven't tried the current version of splunkforwarder.

Tags (1)

wrangler2x
Motivator

The splunk documentation for inputs.conf says, "A file whose modtime falls outside this time window when seen for the first time will not be indexed at all." So, if you had set ignoreOlderThan=1d on an input then the first time the forwarder looked at the directory it is set to watch for that input it would ignore all log files older than one day, and further even if their mod time changes to the current day they will still be ignored.

As for your item #2 using 60s... Sounds like the file you created should have been ignored, unless something updated it within the 60s window!

0 Karma

somesoni2
Revered Legend

It would be (and should have been) a nice feature. We have test environments with data is not so frequent and we do lose some data frequently due to this.

0 Karma

cramasta
Builder

From how I understand the setting to work (im still on 4.3 UF) it checks the modtime during startup of the splunk service. Would be nice for it to periodically.

0 Karma

somesoni2
Revered Legend

It's happening with UF 5.0.5 version as well. Is this a bug and if yes, is it fixed in any release after 5.0.5?

0 Karma
Get Updates on the Splunk Community!

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...