topic Re: Does Splunk recognize when buckets are deleted? in Deployment Architecture

Does Splunk recognize when buckets are deleted?

lycollicott — Mon, 06 Jun 2016 14:56:54 GMT

I am doing a simple recovery test and deleted some warm buckets, but Splunk doesn't seem to even realize anything is wrong. Is this normal?

Re: Does Splunk recognize when buckets are deleted?

jkat54 — Mon, 06 Jun 2016 14:59:40 GMT

Splunk will perform bucket fixups periodically and find the bucket no longer exists. At which time it will log a message or two or three and then remove the bucket from the manifest.

See index=_internal log_level=warn* OR log_level=err*. The events should occur in less than 24 hours after the manual removal. Searches will just have "holes" in the data if a searchable copy of the bucket doesnt exist.

Re: Does Splunk recognize when buckets are deleted?

lycollicott — Mon, 06 Jun 2016 16:27:35 GMT

That search was the first thing I checked, but it had nothing about these buckets.

Re: Does Splunk recognize when buckets are deleted?

jkat54 — Mon, 06 Jun 2016 16:31:03 GMT

Probably has to with the log verbosity on BucketMover or something. I'd file a low priority ticket with support if you're THAT interested.

Re: Does Splunk recognize when buckets are deleted?

splunk_force_as — Mon, 06 Jun 2016 17:46:55 GMT

What does |dbinspect index= return for these buckets? Splunk should eventually log an error message since there should be metadata associated with the deleted buckets, but you will have data gaps since the raw data is deleted.

Re: Does Splunk recognize when buckets are deleted?

lycollicott — Mon, 06 Jun 2016 17:58:57 GMT

I deleted bucket ids 32-34 and 37-39 and dbinspect only shows results for 35-36, so it is still unaware that anything is missing.

Re: Does Splunk recognize when buckets are deleted?

lycollicott — Tue, 07 Jun 2016 19:29:50 GMT

From Splunk Support:

"Splunk assumes that you are doing this on-purpose and therefore would not send any WARN/ERROR events.

The only reason you would alert is if a bucket were corrupt or never made it. Once it was there and you deleted it, from Splunk's perspective, everything was functioning."

That doesn't seem like a sound process to me, but that's the explanation thus far.

UPDATE June 14: I've done some more testing and I just can't accept the outcome. Splunk is essentially a database and as an old
Oracle DBA I would expect/assume that it has some self-awareness of its integrity. I'm going to ask for this to be considered a defect.

UPDATE June 30: Support is going to perform some testing and submit an enhancement request.

UPDATE July 06: Support submitted enhancement request SPL-123789