I have read about the "delete" command and used it. However, my security people want certain events gone without the possibility of recovery. I've looked a little at CLI Search with -output table that looks promising. The idea would be to export the index, remove the offending data and re import/index the result. The original source is long gone. Has anybody had to attempt anything similar?
I found a way that worked for me.
• suspend the offending data feed
• run a search that returns the offending data and pipe to |delete
• Take the index offline
• backup the index
• run a shell script (I'm not much of a script jocky) that returns buckets that have "deletes" folders under rawdata
o pass that bucket name to splunk's exporttool and output as a -csv workfile
o pass the csv workfile output to splunk's importtool and create/output a new bucket with the original name "bucketname.new"
o remove the old bucket and rename bucketname.new to bucketname
• put the index back online/test
• resume the data feed
It is a lot of steps, in my case it took 10 hours to complete (mainly waiting on the export/import to finish) and I had to process warm and cold buckets on 12 index peers. I ran these as background tasks. One for warm buckets. One for cold buckets. I performed a lot of tests before turning this loose. In the end, all the data that had been |delete(d) was gone. Since there was about 5 years of history in play, worth the effort.
One option would be to use the
dump command along with the
clean command. After which you would re-index the events.
You would essentially execute a search that identifies the good events and dump them to local disk in raw format. See Splunk dump command in Search Reference.
You would then
clean the index of all events via the
splunk clean ... CLI command. See Remove data from one or all indexes.
Finally, you would re-index the events that were dumped to disk.
delete search command only marks data for exclusion in subsequent searches.
If you want to remove specific data, you should use the
clean CLI command.
If you want to remove an index entirely, use the
remove index CLI command.
See Remove indexes and indexed data in the Managing Indexers and Clusters of Indexers manual.