All Apps and Add-ons

Can Splunk remove events before data input?

tamduong16
Contributor

I make Splunk monitors my directory and in this directory, I have lots of files. I do realize that every time I erase all content in a particular file and then rewrite it with new information, Splunk indexes new information in but I can't figure out a way to make Splunk delete old event. Is there a way for me to automatically set Splunk to delete all events in that index before it indexes new data in? And most importantly, could I do this without stopping splunk? Thanks

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey, this isn't really possible. Splunk has no real data deletion feature, besides the | delete command, that a) only makes data invisible, but does not really delete it, and b) has to be called manually with the data that is to be "deleted".

Other than that, there is only the retention time, that can be configured per index, although it's not super precise in what it deletes (you might have older events still available). But neither this nor the delete command can be initiated by a data input.

Alternatively, if you can put your data in CSV format, you could put it in a lookup file instead if indexing it - that way you would only have the current state of the file available as data in Splunk.

View solution in original post

xpac
SplunkTrust
SplunkTrust

Hey, this isn't really possible. Splunk has no real data deletion feature, besides the | delete command, that a) only makes data invisible, but does not really delete it, and b) has to be called manually with the data that is to be "deleted".

Other than that, there is only the retention time, that can be configured per index, although it's not super precise in what it deletes (you might have older events still available). But neither this nor the delete command can be initiated by a data input.

Alternatively, if you can put your data in CSV format, you could put it in a lookup file instead if indexing it - that way you would only have the current state of the file available as data in Splunk.

tamduong16
Contributor

This's a great tip. thank you

0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...