Getting Data In

Why is batch processing not removing files after indexing them in Splunk 6.2.1?

jizzmaster
Path Finder

I have an app that is not removing/deleting the files after consuming them. They are indexed appropriately, but just not deleted afterwards.

inputs.conf

[batch:///opt/splunk/etc/apps/my-special-app/pickup/*.json]
index = test
sourcetype = nessus_json
move_policy = sinkhole

I have tested this on a second Splunk box and the exact same app will correctly remove the files after indexing them. I can't tell where the issue may be on this main Splunk box, however. Any suggestions?

On Splunk v6.2.1. This worked a month or so ago. I'd rather figure out the cause before moving to upgrade the Splunk instance.

0 Karma

micahkemp
Champion

I forced a permissions issue with a file such that the Splunk user had read, not write permissions to a file that was configured as a batch input. It resulted in this line in splunkd.log:

11-24-2017 22:49:10.062 +0000 ERROR TailReader - Unable to remove sinkhole file: path=/tmp/batch_del_fail.log, errno=Operation not permitted

Can you look for a similar message to verify if it is a simple permissions issue?

0 Karma

samhays
Path Finder

Sounds like the permissions on the filesystem got screwed up. Check to make sure the account splunk is running as can actually delete the files.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...