All Apps and Add-ons

Why does Splunk forwarder oneshot continues to forward file data even when file is deleted?

othersider2
New Member

I've been running some tests with splunk forwarder oneshot and noticed that if I issue a splunk forwarder one shot CLI command on a huge file (3 gb), the command finishes instantly. Then I immediately delete the file. Much to my surprise, splunk forwarder still is able to send the file data to the server receiver. I thought it would complain that the file was missing since I deleted it.

What is going on behind the scenes here?

BTW, the reason I'm asking is because I am have a requirement to delete files after calling oneshot, but I want to make sure that the files are completely received by the server prior to deleting. If someone could help me with that, much appreciated.

Labels (1)
Tags (1)
0 Karma
1 Solution

dkeck
Influencer

Hi,

sounds like your data might be queue, but if you want the file to be deleted after ingestion, use a batch input and move_policy = sinkhole. So you don´t have to do it manually

[batch://<path>]
 disabled = false
 index = 
 sourcetype =
move_policy = sinkhole

View solution in original post

0 Karma

aokur_splunk
Splunk Employee
Splunk Employee

one reason why you might see this is because the file system maintains a lock on a file if it is in use by a current process. so while you may have issued your file delete command, since splunk was actively using it, the file descriptor may not release the file - splunk will continue ingesting it

The option to sinkhole/batch process mentioned is a good way though of deleting the file after ingest

0 Karma

dkeck
Influencer

Hi,

sounds like your data might be queue, but if you want the file to be deleted after ingestion, use a batch input and move_policy = sinkhole. So you don´t have to do it manually

[batch://<path>]
 disabled = false
 index = 
 sourcetype =
move_policy = sinkhole
0 Karma

VijaySrrie
Builder

Hi @dkeck 

What is the use of 

move_policy = sinkhole

 

And In which scenario we go for batch ? 

0 Karma

dkeck
Influencer

Any luck with that? If it was helpfull please accept the answer, thank you 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...