Getting Data In

Monitor vs Batch Job for inputting data

JoeSco27
Communicator

Currently, my preProd environment is set up to monitor logs from 100-150 servers with the monitor stanza in inputs.conf. I have been asked to research changing all these stanza to batch jobs because the pre-Prod forwarders apparently stop running from an overload of the saved files on them. I was wondering if anyone has had to tackle this before or if there is a better way of doing this. I figure the forwarder admins don't want to set up jobs on their own servers so they want Splunk to delete the files once indexed.

Assistance would be appreciated. Thank you

0 Karma
1 Solution

krish3
Contributor

Usually Batch input is used for large historical data. It is not a good practice for small and continuously updating files.

Consider a case where i faced using batch input i fed splunk a large files and folders and in between the license limit was reached and stopped splunk to wait 24 hours and later i couldn't start indexing from where it stopped because the file was deleted. Due to move_policy = sinkhole policy that is mandatory for batch inputs. And I had no clue till what point the data was indexed all i did was i flushed the index and reindex whole data using monitor.

It depends on the size of the data and its availability you should be able to choose the best one.

View solution in original post

krish3
Contributor

Usually Batch input is used for large historical data. It is not a good practice for small and continuously updating files.

Consider a case where i faced using batch input i fed splunk a large files and folders and in between the license limit was reached and stopped splunk to wait 24 hours and later i couldn't start indexing from where it stopped because the file was deleted. Due to move_policy = sinkhole policy that is mandatory for batch inputs. And I had no clue till what point the data was indexed all i did was i flushed the index and reindex whole data using monitor.

It depends on the size of the data and its availability you should be able to choose the best one.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...