topic Re: How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day? in Getting Data In

How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

vaibhavagg2006 — Fri, 05 Feb 2016 23:37:52 GMT

I am monitoring a folder which has high level of nesting and daily, 1000's of folders gets created. The name of the folder is unique based on some id. I am seeing a delay of 10-12 hours in getting the logs which are placed deep in the nth folder. I believe this is because Splunk checks for each and every folder sequentially for a match. Can we ignore folders older than 1 day so that Splunk does not search inside old folders? I am using a universal forwarder with good bunch of indexers to index the data. There is no throughput issue. The daily ingestion is around 1-2 gigs.
Below is my inputs.conf stanza

[monitor:///<folder path>]
_TCP_ROUTING = prod
ignoreOlderThan = 2d
whitelist = .log
index = index1
sourcetype = sample_sourcetype
disabled = 0

Please provide your inputs on this issue.

Re: How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

ddrillic — Sat, 06 Feb 2016 01:19:28 GMT

http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Monitorfilesanddirectorieswithinputs.conf covers it.
ignoreOlderThan = 2d seems to be the right set-up.

Re: How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

vaibhavagg2006 — Mon, 08 Feb 2016 18:15:07 GMT

I believe "ignoreOlderThan" will only ignore files. My problem is splunk is taking too much time in traversing through the folders to find a match.