Can I undo "| delete"?

[arichman]

So, I ran | delete on the wrong query. ooopz.
Per the various docs I've poured through, it seems that warm directories are where I would find events that I want to thaw.

Is there any advice out there to offer for rebuilding indexes because of events marked for delete ? I would move them all to the thaw bucket and rebuild but I fear I will have lots of duplicates in that case since I only ran the delete cmd on a certain subset of events. Any suggestions are appreciated!

Thanks
Adam

[nspencer16]

Updated for version 8.x;

Splunk now creates splunk-autogen-params.dat .  If  Splunk is stopped and these files are removed, the | deleted data will become searchable again.  

[dxu_splunk · Accepted solution]

| delete actually creates a deletes directory on the buckets directories that serve to filter out events from the search. If you only ran | delete once, then you can just stop your indexer(s), manually delete all the deletes directories in the buckets, and then restart them back up.

If you have called | delete for other items as well, this wouldn't work since you don't know which folder correspond to which delete, assuming you want to keep the previous deletes.

The folders to look for is at $SPLUNK_DB/<index>/db/<bucket_id>/deletes/.

[armesh]

Tried in Splunk 8.1.3

Removed the   ../<indexname>/db/<bucket_id>/rawdata/deletes folder.

Didn't work. 

[lguinn2]

You can't "thaw" deleted events. You can't undelete them, either.

If you clean or rebuild the buckets, there will be absolutely no way to recover the deleted data.

However, the Splunk documentation says that the events are marked as deleted, but that the disk space is not recovered.
I don't know exactly what "marked as deleted" means, but perhaps it means that events (or partial events) could be recovered.

If this is not production data, it probably isn't worth the effort to recover the deleted data, even if it is possible.

If it is production data, then you might contact Splunk Support and see if they have anything to offer. I expect that - if anything - you will be doing some risky manual work on production data. And maybe running the rebuild command on the buckets as well - after you have repaired them.

Are your indexers in a cluster? If yes, then your problem is much, much worse. I doubt you will be able to recover anything. You will may be better off trying to sort out the inputs and selectively re-index them. And you still will probably have duplicate events.

[arichman]

the indexers are in a cluster, but an index rebuild among clustered indexers is not an impossible feat, according to docs. fwiw, the events all belong to a single index.

Anybody else have real knowledge of the nature of events "marked for delete"? are they only removed from the bloomfilter? would a rebuild re-include them?

[lguinn2]

A rebuild will not re-include them, it will recover the space they were occupying. Same thing as when you empty the trash on your PC. You can rebuild among clustered indexers, of course, but that will not solve your problem.