Getting Data In

inputs.conf using batch in stanza

ben_leung
Builder

If my forwarder's inputs.conf stanza line is set to batch instead of monitor, it will delete the file after it is indexed.

Lets say my inputs.conf stanza is this

[batch:///home/bleung/tutorialdata]
disabled = false
index = tutorial
sourcetype = testing

If the files there are deleted in that path and I put in a file there with the same data just different timestamps, will the forwarder still send data to the indexer?

Scenario:
I scp a file to the forward server and is oneshot'ed. Next day I scp a file with same data just different timestamps. The indexer does not receive any data for that day.

Splunk version 5+
Splunk forwarder version 5+

Tags (3)
1 Solution

dmlee
Communicator

according to the description of "batch" in inputs.conf.spec , you should set move_policy = sinkhole .

move_policy = sinkhole
* IMPORTANT: This attribute/value pair is required. You *must* include "move_policy = sinkhole" when defining batch inputs.
* This loads the file destructively.  
* Do not use the batch input type for files you do not want to consume destructively.
* As long as this is set, Splunk won't keep track of indexed files. Without the "move_policy = sinkhole" setting,  it won't load the files destructively and will keep a track of them.

View solution in original post

dmlee
Communicator

according to the description of "batch" in inputs.conf.spec , you should set move_policy = sinkhole .

move_policy = sinkhole
* IMPORTANT: This attribute/value pair is required. You *must* include "move_policy = sinkhole" when defining batch inputs.
* This loads the file destructively.  
* Do not use the batch input type for files you do not want to consume destructively.
* As long as this is set, Splunk won't keep track of indexed files. Without the "move_policy = sinkhole" setting,  it won't load the files destructively and will keep a track of them.
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...