Getting Data In

How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

vaibhavagg2006
Communicator

Hi

I am monitoring a folder which has high level of nesting and daily, 1000's of folders gets created. The name of the folder is unique based on some id. I am seeing a delay of 10-12 hours in getting the logs which are placed deep in the nth folder. I believe this is because Splunk checks for each and every folder sequentially for a match. Can we ignore folders older than 1 day so that Splunk does not search inside old folders? I am using a universal forwarder with good bunch of indexers to index the data. There is no throughput issue. The daily ingestion is around 1-2 gigs.
Below is my inputs.conf stanza

[monitor:///<folder path>]
_TCP_ROUTING = prod
ignoreOlderThan = 2d
whitelist = .log
index = index1
sourcetype = sample_sourcetype
disabled = 0

Please provide your inputs on this issue.

0 Karma

ddrillic
Ultra Champion

http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Monitorfilesanddirectorieswithinputs.conf covers it.
ignoreOlderThan = 2d seems to be the right set-up.

0 Karma

vaibhavagg2006
Communicator

I believe "ignoreOlderThan" will only ignore files. My problem is splunk is taking too much time in traversing through the folders to find a match.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...