Why do soft deleted sources return after indexer restart?

[jaredlaney]

Why do soft deleted sources return after indexer restart? This has happened to us every time. We are performing a high number of soft deletes.

[s2_splunk]

I cannot reproduce this on my standalone instance. However, I did find an open bug which describes your symptoms when using |delete in an indexer cluster (SPL-100516).

Are you using a clustered deployment?

[jaredlaney]

Yes, we are using a clustered deployment.

[jaredlaney]

I can't access this bug. Is there anyway you could send me a quick explanation on it?

[s2_splunk]

All I can provide you is the bug description: Events deleted in an index cluster via the "| delete" search operator reappear after cluster restart

If you are a Splunk customer with a support entitlement, please open a support case for this, so your case# can be added to the bug ticket.

[jaredlaney]

Ok, thanks. Will do.

[s2_splunk]

No problem. It may be worthwhile thinking about a different approach to solving your use case. As you may know, | delete does not physically delete events, it just prevents them from being searchable.
Maybe you can configure your index retention settings such that old data ages out according to your needs.
Or use tags to flag outdated events and modify your searches to not include tagged events, if you cannot reliably use _time to limit your search results to the latest data.
Just a thought.

[jaredlaney]

We are using frozentimeperiodinsecs.

We are not using tags to flag outdated data. Do you have a good reference?

[s2_splunk]

Sorry, on second thought, using tags is probably not going to work well for this, unless you have a single field value in your dataset that is common to all events you need to hide. For example, if you can use a date field, you could tag all events from a specific date as "outdated" and include something like NOT "tag::date=outdated to your searches.

[s2_splunk]

Can you elaborate a bit, please? What's a "soft deleted source"? Can you describe in more detail what you are trying to do and what symptoms you are seeing?

[jaredlaney]

@ssievert - Any ideas on this?

[jaredlaney]

Sure. we pipe to delete quite often. Like the following:

index=index1 | delete

We consider this to be the fake or soft delete compared to the cli index truncate

[s2_splunk]

OK, so you are saying that when you are doing a | delete and restart your indexer, the events that were subject to deletion are searchable again?
What exact version of Splunk are you running?

[jaredlaney]

We're running 6.2.2. Yes, we have a series of what we call snapshot indexes where we delete the data daily and re-ingest.

Yes, old source files reappear and are searchable when we restart our indexers.