topic Re: Why do soft deleted sources return after indexer restart? in Getting Data In

Why do soft deleted sources return after indexer restart?

jaredlaney — Tue, 10 Nov 2015 18:39:22 GMT

Why do soft deleted sources return after indexer restart? This has happened to us every time. We are performing a high number of soft deletes.

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Tue, 10 Nov 2015 21:23:12 GMT

Can you elaborate a bit, please? What's a "soft deleted source"? Can you describe in more detail what you are trying to do and what symptoms you are seeing?

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Tue, 10 Nov 2015 21:52:43 GMT

Sure. we pipe to delete quite often. Like the following:

index=index1 | delete

We consider this to be the fake or soft delete compared to the cli index truncate

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Wed, 11 Nov 2015 01:44:36 GMT

OK, so you are saying that when you are doing a | delete and restart your indexer, the events that were subject to deletion are searchable again?
What exact version of Splunk are you running?

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 13:35:01 GMT

@ssievert - Any ideas on this?

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 17:31:48 GMT

We're running 6.2.2. Yes, we have a series of what we call snapshot indexes where we delete the data daily and re-ingest.

Yes, old source files reappear and are searchable when we restart our indexers.

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Thu, 12 Nov 2015 18:49:18 GMT

I cannot reproduce this on my standalone instance. However, I did find an open bug which describes your symptoms when using |delete in an indexer cluster (SPL-100516).

Are you using a clustered deployment?

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 18:57:52 GMT

Yes, we are using a clustered deployment.

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 19:00:11 GMT

I can't access this bug. Is there anyway you could send me a quick explanation on it?

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Thu, 12 Nov 2015 19:21:03 GMT

All I can provide you is the bug description: Events deleted in an index cluster via the "| delete" search operator reappear after cluster restart

If you are a Splunk customer with a support entitlement, please open a support case for this, so your case# can be added to the bug ticket.

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 19:24:09 GMT

Ok, thanks. Will do.

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Thu, 12 Nov 2015 19:29:20 GMT

No problem. It may be worthwhile thinking about a different approach to solving your use case. As you may know, | delete does not physically delete events, it just prevents them from being searchable.
Maybe you can configure your index retention settings such that old data ages out according to your needs.
Or use tags to flag outdated events and modify your searches to not include tagged events, if you cannot reliably use _time to limit your search results to the latest data.
Just a thought.

Re: Why do soft deleted sources return after indexer restart?

jaredlaney — Thu, 12 Nov 2015 19:34:03 GMT

We are using frozentimeperiodinsecs.

We are not using tags to flag outdated data. Do you have a good reference?

Re: Why do soft deleted sources return after indexer restart?

s2_splunk — Fri, 13 Nov 2015 17:37:42 GMT

Sorry, on second thought, using tags is probably not going to work well for this, unless you have a single field value in your dataset that is common to all events you need to hide. For example, if you can use a date field, you could tag all events from a specific date as "outdated" and include something like NOT "tag::date=outdated to your searches.