About delete command error.

[yutaka1005]

In my environment SH, indexer 1, indexer 2 exist, and distributed search is done for indexers 1 and 2 from SH.

Yesterday, since data was duplicated in indexers 1 and 2, I give can_delete role to admin user of SH, and executed delete command on SH.
However, despite all the data of indexer 1 being displayed as "deleted", all the data of indexer 2 was "errors".
Also, the following error message appeared.

["hostname of indexer 2"] You do not have the capability to delete from "index name"

However, when I executed the same command again, all the data of indexer 2 was "deleted" this time.

I thought that connection of SH between indexer 2 was being disconnected when I executed delete command and received error messages,
but there was no error that communication with the indexer 2 was interrupted.

What is this phenomenon?
It will be very helpful if someone tells me.

[davpx · Accepted solution]

The reason this most likely failed on the 2nd indexer is that your authorization change may have yet to propagate to that indexer. By the time you tried again, the change made its way to indexer 2's bundle.

A bit off topic but may be related, how big is your search bundle and are you seeing messages for it taking longer than expected to replicate?

[yutaka1005]

Thank you for answer!

The first sentence implies that propagation of information that "can_delete" was given to the "admin" user was delayed?

Excuse me
What does the second sentence mean?

[davpx]

A search bundle is basically a copy of your search head configurations which are replicated to your indexer. If you have a lot of apps or some that contain large files, it can sometimes take awhile to replicate.

[yutaka1005]

I got it.

In my environment, there are not such many apps.
So I do not think there will be a long time to replicate.

[nickhills]

Did you see any errors in the splunkd or webservice log?

Since re-running it worked correctly, I would be inclined to think it a transient error - but any logs indicating network connectivity issues would give some comfort that the issue was further down the stack than Splunk.

[yutaka1005]

No, I looked for it but I could not find it.

I thought it is transient error too, but I can not find anything that can prove it ...

[inventsekar]

hope the indexers are not clustered, right.

One more thing....were you trying to delete data from real time search?

Note: You cannot run the delete command during a real-time search; you cannot delete events as they come in. If you try to use delete during a real-time search, Splunk Enterprise will display an error.

[yutaka1005]

Thank you for comment!

First, my indexers are not clustered.

Second, I excuted delete with the specified search period not real-time search.

[inventsekar]

Ohk sure. So, the issue is that, when you ran first, you got error.. on second run, delete worked fine. Right?

Is it a repeating issue? Try to rerun delete command for some sample logs (careful, data deleted is irreversible, ..once deleted, data can't be retrieved back)

[yutaka1005]

Yes, you right.

No it is first time.
I want to retry, but I can not easily take sample logs and delete it...