We have a v6.1.6 Windows server 2008 distributed Splunk environment.
On the Indexers the following event is being written to splunkd.log every minute:
WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user
What could be causing this and how can I make this go away?
Thanks in advance for your help
This happens when you keep cluster master in maintenance mode and re-add peer to cluster. Always ensure you keep cluster master out of maintenance mode when you re-add peer.
You can simply fix this by going to splunkweb on Search head _ settings_distributed Search_ search peers . Select the peer which is having issues and add the admin user/password _ save
You can fix this by deleting the search head key and directory from the indexer that is reporting the issue.
On indexer only delete just the search head that is having the issue.
splunk index> rm -rf /opt/splunk/etc/auth/distServerKeys/mysearchheadhost
On the search head readd the indexer via the gui.
This will recreate the directory on the indexer and resend the key across to the index and it should fix this issue.
I deleted the folder $SPLUNK_HOME/etc/auth/distServerKeys/mysearchheadhost. I did not see a message saying that replication had failed for the indexer. I entered the admin/password and saved but the warning is still there. The warning occurs once a minute on each indexer.
Mysearchheadhost should be replaced with the name of your actual search head!!! There should be a few directories in there so have a look first (don't cut and paste the command as I posted it)
mysearchheadhost was replaced with the name of the search head. This is a Windows instance so I did not use your command. I deleted the "mysearchheadhost" directory then I logged in to splunkweb on the search head and re-entered the admin/password details for the indexer in the distributed search peers view. Note, I did not delete the search peer, I just re-entered to admin/password details and "mysearchheadhost" directory re-appeared on the indexer. What is the name of the key that must be deleted? The only key found in "mysearchheadhost" directory is trusted.pem
You shouldn't need to readd it as it should still be listed on the search head (you removed it from the indexer only remember). All you need to do is re-auth it. You should see that the replication has failed for that particular search peer. Just click on its name and put in the remote admin/password and click save.
This will recreate the directory on the indexer and copy across the public key from the search head.
I did this for 2 indexers today hence me stumbling on this un answered question when I was looking for the solution 🙂