Getting Data In
Highlighted

About delete command error.

Builder

In my environment SH, indexer 1, indexer 2 exist, and distributed search is done for indexers 1 and 2 from SH.

Yesterday, since data was duplicated in indexers 1 and 2, I give can_delete role to admin user of SH, and executed delete command on SH.
However, despite all the data of indexer 1 being displayed as "deleted", all the data of indexer 2 was "errors".
Also, the following error message appeared.

["hostname of indexer 2"] You do not have the capability to delete from "index name"

However, when I executed the same command again, all the data of indexer 2 was "deleted" this time.

I thought that connection of SH between indexer 2 was being disconnected when I executed delete command and received error messages,
but there was no error that communication with the indexer 2 was interrupted.

What is this phenomenon?
It will be very helpful if someone tells me.

0 Karma
Highlighted

Re: About delete command error.

Champion

hope the indexers are not clustered, right.

One more thing....were you trying to delete data from real time search?

Note: You cannot run the delete command during a real-time search; you cannot delete events as they come in. If you try to use delete during a real-time search, Splunk Enterprise will display an error.

0 Karma
Highlighted

Re: About delete command error.

Builder

Thank you for comment!

First, my indexers are not clustered.

Second, I excuted delete with the specified search period not real-time search.

0 Karma
Highlighted

Re: About delete command error.

Champion

Ohk sure. So, the issue is that, when you ran first, you got error.. on second run, delete worked fine. Right?

Is it a repeating issue? Try to rerun delete command for some sample logs (careful, data deleted is irreversible, ..once deleted, data can't be retrieved back)

0 Karma
Highlighted

Re: About delete command error.

Builder

Yes, you right.

No it is first time.
I want to retry, but I can not easily take sample logs and delete it...

0 Karma
Highlighted

Re: About delete command error.

Ultra Champion

Did you see any errors in the splunkd or webservice log?

Since re-running it worked correctly, I would be inclined to think it a transient error - but any logs indicating network connectivity issues would give some comfort that the issue was further down the stack than Splunk.

0 Karma
Highlighted

Re: About delete command error.

Builder

No, I looked for it but I could not find it.

I thought it is transient error too, but I can not find anything that can prove it ...

0 Karma
Highlighted

Re: About delete command error.

Communicator

The reason this most likely failed on the 2nd indexer is that your authorization change may have yet to propagate to that indexer. By the time you tried again, the change made its way to indexer 2's bundle.

A bit off topic but may be related, how big is your search bundle and are you seeing messages for it taking longer than expected to replicate?

View solution in original post

Highlighted

Re: About delete command error.

Builder

Thank you for answer!

The first sentence implies that propagation of information that "can_delete" was given to the "admin" user was delayed?

Excuse me
What does the second sentence mean?

0 Karma
Highlighted

Re: About delete command error.

Communicator

A search bundle is basically a copy of your search head configurations which are replicated to your indexer. If you have a lot of apps or some that contain large files, it can sometimes take awhile to replicate.

0 Karma