Monitor vs Batch Job for inputting data

[JoeSco27]

Currently, my preProd environment is set up to monitor logs from 100-150 servers with the monitor stanza in inputs.conf. I have been asked to research changing all these stanza to batch jobs because the pre-Prod forwarders apparently stop running from an overload of the saved files on them. I was wondering if anyone has had to tackle this before or if there is a better way of doing this. I figure the forwarder admins don't want to set up jobs on their own servers so they want Splunk to delete the files once indexed.

Assistance would be appreciated. Thank you

[krish3 · Accepted solution]

Usually Batch input is used for large historical data. It is not a good practice for small and continuously updating files.

Consider a case where i faced using batch input i fed splunk a large files and folders and in between the license limit was reached and stopped splunk to wait 24 hours and later i couldn't start indexing from where it stopped because the file was deleted. Due to move_policy = sinkhole policy that is mandatory for batch inputs. And I had no clue till what point the data was indexed all i did was i flushed the index and reindex whole data using monitor.

It depends on the size of the data and its availability you should be able to choose the best one.