topic Does Splunk ingest files that existed before the remote folder monitor was created? in Getting Data In

Does Splunk ingest files that existed before the remote folder monitor was created?

williamcharlton — Thu, 10 Oct 2019 14:57:48 GMT

I have a client server with a universal forwarder configured to forward data to an index server. On the client server, I have a folder "X" full of CSV files. If I create a remote folder monitor for the client server folder "X" on my deployment server and deploy it to the client server.

Will Splunk process the CSV files that are already there. or will Splunk not do anything until the folder contents change?

Re: Does Splunk ingest files that existed before the remote folder monitor was created?

gcusello — Thu, 10 Oct 2019 15:07:44 GMT

Hi williamcharlton0028,
Splunk will process all events the it can find in the monitored folder from 6 years old.
Then it will process only changes.

If you don't want too old data, you can exclude older data using the option

ignoreOlderThan = <non-negative integer>[s|m|h|d]

in inputs.conf stanza.

Bye.
Giuseppe

Re: Does Splunk ingest files that existed before the remote folder monitor was created?

woodcock — Mon, 14 Oct 2019 23:01:31 GMT

It will forward every file that is there. If any file has events with timestamps that are 6 years old or older, they will all be processed and forwarded but then dropped by the indexers, unless you modify MAX_DAYS_AGO setting. The ignoreOlderThan settings does not work that way that most people expect it to so I would not use it. Instead, if you would like to avoid indexing files, move the old ones to not forward into another directory that is not monitored.