Hi
I am monitoring a folder which has high level of nesting and daily, 1000's of folders gets created. The name of the folder is unique based on some id. I am seeing a delay of 10-12 hours in getting the logs which are placed deep in the nth folder. I believe this is because Splunk checks for each and every folder sequentially for a match. Can we ignore folders older than 1 day so that Splunk does not search inside old folders? I am using a universal forwarder with good bunch of indexers to index the data. There is no throughput issue. The daily ingestion is around 1-2 gigs.
Below is my inputs.conf stanza
[monitor:///<folder path>]
_TCP_ROUTING = prod
ignoreOlderThan = 2d
whitelist = .log
index = index1
sourcetype = sample_sourcetype
disabled = 0
Please provide your inputs on this issue.
http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Monitorfilesanddirectorieswithinputs.conf covers it.
ignoreOlderThan = 2d seems to be the right set-up.
I believe "ignoreOlderThan" will only ignore files. My problem is splunk is taking too much time in traversing through the folders to find a match.