Getting Data In

Monitor vs Batch Job for inputting data

JoeSco27
Communicator

Currently, my preProd environment is set up to monitor logs from 100-150 servers with the monitor stanza in inputs.conf. I have been asked to research changing all these stanza to batch jobs because the pre-Prod forwarders apparently stop running from an overload of the saved files on them. I was wondering if anyone has had to tackle this before or if there is a better way of doing this. I figure the forwarder admins don't want to set up jobs on their own servers so they want Splunk to delete the files once indexed.

Assistance would be appreciated. Thank you

0 Karma
1 Solution

krish3
Contributor

Usually Batch input is used for large historical data. It is not a good practice for small and continuously updating files.

Consider a case where i faced using batch input i fed splunk a large files and folders and in between the license limit was reached and stopped splunk to wait 24 hours and later i couldn't start indexing from where it stopped because the file was deleted. Due to move_policy = sinkhole policy that is mandatory for batch inputs. And I had no clue till what point the data was indexed all i did was i flushed the index and reindex whole data using monitor.

It depends on the size of the data and its availability you should be able to choose the best one.

View solution in original post

krish3
Contributor

Usually Batch input is used for large historical data. It is not a good practice for small and continuously updating files.

Consider a case where i faced using batch input i fed splunk a large files and folders and in between the license limit was reached and stopped splunk to wait 24 hours and later i couldn't start indexing from where it stopped because the file was deleted. Due to move_policy = sinkhole policy that is mandatory for batch inputs. And I had no clue till what point the data was indexed all i did was i flushed the index and reindex whole data using monitor.

It depends on the size of the data and its availability you should be able to choose the best one.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...