Getting Data In

Why is batch processing not removing files after indexing them in Splunk 6.2.1?

jizzmaster
Path Finder

I have an app that is not removing/deleting the files after consuming them. They are indexed appropriately, but just not deleted afterwards.

inputs.conf

[batch:///opt/splunk/etc/apps/my-special-app/pickup/*.json]
index = test
sourcetype = nessus_json
move_policy = sinkhole

I have tested this on a second Splunk box and the exact same app will correctly remove the files after indexing them. I can't tell where the issue may be on this main Splunk box, however. Any suggestions?

On Splunk v6.2.1. This worked a month or so ago. I'd rather figure out the cause before moving to upgrade the Splunk instance.

0 Karma

micahkemp
Champion

I forced a permissions issue with a file such that the Splunk user had read, not write permissions to a file that was configured as a batch input. It resulted in this line in splunkd.log:

11-24-2017 22:49:10.062 +0000 ERROR TailReader - Unable to remove sinkhole file: path=/tmp/batch_del_fail.log, errno=Operation not permitted

Can you look for a similar message to verify if it is a simple permissions issue?

0 Karma

samhays
Path Finder

Sounds like the permissions on the filesystem got screwed up. Check to make sure the account splunk is running as can actually delete the files.

0 Karma
Get Updates on the Splunk Community!

Splunk App for Anomaly Detection End of Life Announcement

Q: What is happening to the Splunk App for Anomaly Detection?A: Splunk is officially announcing the ...

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...