I've been running some tests with splunk forwarder oneshot and noticed that if I issue a splunk forwarder one shot CLI command on a huge file (3 gb), the command finishes instantly. Then I immediately delete the file. Much to my surprise, splunk forwarder still is able to send the file data to the server receiver. I thought it would complain that the file was missing since I deleted it.
What is going on behind the scenes here?
BTW, the reason I'm asking is because I am have a requirement to delete files after calling oneshot, but I want to make sure that the files are completely received by the server prior to deleting. If someone could help me with that, much appreciated.
Hi,
sounds like your data might be queue, but if you want the file to be deleted after ingestion, use a batch input and move_policy = sinkhole
. So you don´t have to do it manually
[batch://<path>]
disabled = false
index =
sourcetype =
move_policy = sinkhole
one reason why you might see this is because the file system maintains a lock on a file if it is in use by a current process. so while you may have issued your file delete command, since splunk was actively using it, the file descriptor may not release the file - splunk will continue ingesting it
The option to sinkhole/batch process mentioned is a good way though of deleting the file after ingest
Hi,
sounds like your data might be queue, but if you want the file to be deleted after ingestion, use a batch input and move_policy = sinkhole
. So you don´t have to do it manually
[batch://<path>]
disabled = false
index =
sourcetype =
move_policy = sinkhole
Any luck with that? If it was helpfull please accept the answer, thank you 🙂