topic Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user in Getting Data In

Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Ant1D — Tue, 09 Jun 2015 11:28:35 GMT

Hi,

We have a v6.1.6 Windows server 2008 distributed Splunk environment.
On the Indexers the following event is being written to splunkd.log every minute:

WARN  AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

What could be causing this and how can I make this go away?

Thanks in advance for your help

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Lucas_K — Tue, 23 Jun 2015 23:03:12 GMT

You can fix this by deleting the search head key and directory from the indexer that is reporting the issue.

On indexer only delete just the search head that is having the issue.

splunk index> rm -rf /opt/splunk/etc/auth/distServerKeys/mysearchheadhost

On the search head readd the indexer via the gui.

This will recreate the directory on the indexer and resend the key across to the index and it should fix this issue.

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Ant1D — Wed, 24 Jun 2015 08:19:11 GMT

Hi Lucas, when you say "On the search head readd the indexer via the gui" do you mean add this indexer as a search peer of that search head again?

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Lucas_K — Wed, 24 Jun 2015 09:32:44 GMT

You shouldn't need to readd it as it should still be listed on the search head (you removed it from the indexer only remember). All you need to do is re-auth it. You should see that the replication has failed for that particular search peer. Just click on its name and put in the remote admin/password and click save.

This will recreate the directory on the indexer and copy across the public key from the search head.

I did this for 2 indexers today hence me stumbling on this un answered question when I was looking for the solution 🙂

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Ant1D — Wed, 24 Jun 2015 09:36:14 GMT

OK I will try this when I have a moment and see if it fixes the issue. Thanks

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Ant1D — Mon, 13 Jul 2015 15:52:50 GMT

I deleted the folder $SPLUNK_HOME/etc/auth/distServerKeys/mysearchheadhost. I did not see a message saying that replication had failed for the indexer. I entered the admin/password and saved but the warning is still there. The warning occurs once a minute on each indexer.

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Lucas_K — Tue, 14 Jul 2015 04:48:51 GMT

You haven't deleted the right key if you don't get the replication failure. Double check that you deleted the right one off the indexer.

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Lucas_K — Tue, 14 Jul 2015 04:50:39 GMT

Mysearchheadhost should be replaced with the name of your actual search head!!! There should be a few directories in there so have a look first (don't cut and paste the command as I posted it)

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Ant1D — Tue, 14 Jul 2015 08:44:28 GMT

mysearchheadhost was replaced with the name of the search head. This is a Windows instance so I did not use your command. I deleted the "mysearchheadhost" directory then I logged in to splunkweb on the search head and re-entered the admin/password details for the indexer in the distributed search peers view. Note, I did not delete the search peer, I just re-entered to admin/password details and "mysearchheadhost" directory re-appeared on the indexer. What is the name of the key that must be deleted? The only key found in "mysearchheadhost" directory is trusted.pem

Re: Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

RishiMandal — Sat, 25 Jul 2020 19:15:15 GMT

This happens when you keep cluster master in maintenance mode and re-add peer to cluster. Always ensure you keep cluster master out of maintenance mode when you re-add peer.
You can simply fix this by going to splunkweb on Search head _ settings_distributed Search_ search peers . Select the peer which is having issues and add the admin user/password _ save