We have a large search head cluster that we saw the count of data spike on. Turns out, there's an issue with timestamps going BACKWARDS in time.
I've been creating props for these to clear up the issues and its been working as expected. After fixing, I run my search and pipe to the delete command, something like: index=* latest=-60d@d | delete
I run this command on one of the search heads (delete user is only allowed to log in on one search head), which is not the captain, and then go to Splunk Web and run the search again and nothing is changed.
My question is:
Once I pipe to the delete command, does this not replicate to all search heads? Do I need to re-push config bundles again?
Search head the delete user runs on: index=* latest=-60d@d | delete
0 results found
Search head captain: index=* latest=-60d@d | delete
The data is stored on Indexers and the delete command just makes them un-searchable, so there is nothing special to be done search head cluster as such. How did you provide can_delete role to a user in just one Search Head? Did you update the file (authentication.conf)?