Getting Data In

About delete command error.

yutaka1005
Builder

In my environment SH, indexer 1, indexer 2 exist, and distributed search is done for indexers 1 and 2 from SH.

Yesterday, since data was duplicated in indexers 1 and 2, I give can_delete role to admin user of SH, and executed delete command on SH.
However, despite all the data of indexer 1 being displayed as "deleted", all the data of indexer 2 was "errors".
Also, the following error message appeared.

["hostname of indexer 2"] You do not have the capability to delete from "index name"

However, when I executed the same command again, all the data of indexer 2 was "deleted" this time.

I thought that connection of SH between indexer 2 was being disconnected when I executed delete command and received error messages,
but there was no error that communication with the indexer 2 was interrupted.

What is this phenomenon?
It will be very helpful if someone tells me.

0 Karma
1 Solution

davpx
Communicator

The reason this most likely failed on the 2nd indexer is that your authorization change may have yet to propagate to that indexer. By the time you tried again, the change made its way to indexer 2's bundle.

A bit off topic but may be related, how big is your search bundle and are you seeing messages for it taking longer than expected to replicate?

View solution in original post

davpx
Communicator

The reason this most likely failed on the 2nd indexer is that your authorization change may have yet to propagate to that indexer. By the time you tried again, the change made its way to indexer 2's bundle.

A bit off topic but may be related, how big is your search bundle and are you seeing messages for it taking longer than expected to replicate?

yutaka1005
Builder

Thank you for answer!

The first sentence implies that propagation of information that "can_delete" was given to the "admin" user was delayed?

Excuse me
What does the second sentence mean?

0 Karma

davpx
Communicator

A search bundle is basically a copy of your search head configurations which are replicated to your indexer. If you have a lot of apps or some that contain large files, it can sometimes take awhile to replicate.

0 Karma

yutaka1005
Builder

I got it.

In my environment, there are not such many apps.
So I do not think there will be a long time to replicate.

0 Karma

nickhills
Ultra Champion

Did you see any errors in the splunkd or webservice log?

Since re-running it worked correctly, I would be inclined to think it a transient error - but any logs indicating network connectivity issues would give some comfort that the issue was further down the stack than Splunk.

If my comment helps, please give it a thumbs up!
0 Karma

yutaka1005
Builder

No, I looked for it but I could not find it.

I thought it is transient error too, but I can not find anything that can prove it ...

0 Karma

inventsekar
Ultra Champion

hope the indexers are not clustered, right.

One more thing....were you trying to delete data from real time search?

Note: You cannot run the delete command during a real-time search; you cannot delete events as they come in. If you try to use delete during a real-time search, Splunk Enterprise will display an error.

0 Karma

yutaka1005
Builder

Thank you for comment!

First, my indexers are not clustered.

Second, I excuted delete with the specified search period not real-time search.

0 Karma

inventsekar
Ultra Champion

Ohk sure. So, the issue is that, when you ran first, you got error.. on second run, delete worked fine. Right?

Is it a repeating issue? Try to rerun delete command for some sample logs (careful, data deleted is irreversible, ..once deleted, data can't be retrieved back)

0 Karma

yutaka1005
Builder

Yes, you right.

No it is first time.
I want to retry, but I can not easily take sample logs and delete it...

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...