topic Re: When to use batch input? in Getting Data In

When to use batch input?

dilipvpatel — Mon, 11 Feb 2013 18:06:16 GMT

In the documentation of inputs.conf, Splunk does not recommend to use BATCH input method for small archive sizes and recommends instead to use MONITOR input method.

From Splunk's input.conf documentation:
"NOTE: Batch should only be used for large archives of historic data. If you want to continuously monitor a directory or index small archives, use monitor (see above). Batch reads in the file and indexes it, and then deletes the file from the Splunk instance."

What sized file, Splunk considers it to be large enough so that one can use the BATCH INPUT method? I can understand that one cannot have any hard boundary but I would like to know whether files running in 10s of MBs can be considered large enough or not.

What is the technical reason behind Splunk not allowing to use BATCH method for small archives? How small sized files are better for Monitor input. I know that BATCH method purges the input file once it has indexed/uploaded the events but which performance related reasons are involved for discouraing the use of BATCH input method for small archives.

Re: When to use batch input?

dart — Mon, 11 Feb 2013 21:22:07 GMT

You should use it only when you want your source file to be deleted after indexing.

Otherwise it's the same as monitor

Re: When to use batch input?

dilipvpatel — Mon, 11 Feb 2013 21:58:00 GMT

From Splunk's input.conf documentation: "NOTE: Batch should only be used for large archives of historic data. If you want to continuously monitor a directory or index small archives, use monitor (see above).".....

For small archives, Splunk does not recommend to use batch input. What sized files are considered small archives by Splunk? Small/Large are very subjective terms. What is the Splunk's definition for Small v/s Large archives...

Re: When to use batch input?

dilipvpatel — Mon, 11 Feb 2013 21:58:14 GMT

I want to use Batch input because Splunk will delete source files for me and I would not need to handle their purging. If my system is generating files as small as 1K files, should I use BATCH INPUT. I do not know what this boundary is for small v/s large archives from Splunk's point of view.