Getting Data In

Indexers showing: WARN AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

Motivator

Hi,

We have a v6.1.6 Windows server 2008 distributed Splunk environment.
On the Indexers the following event is being written to splunkd.log every minute:

WARN  AdminHandler:AuthenticationHandler - Denied session token for user: splunk-system-user

What could be causing this and how can I make this go away?

Thanks in advance for your help

0 Karma

Loves-to-Learn

This happens when you keep cluster master in maintenance mode and re-add peer to cluster. Always ensure you keep cluster master out of maintenance mode when you re-add peer.
You can simply fix this by going to splunkweb on Search head _ settings_distributed Search_ search peers . Select the peer which is having issues and add the admin user/password _ save

0 Karma

Motivator

You can fix this by deleting the search head key and directory from the indexer that is reporting the issue.

On indexer only delete just the search head that is having the issue.

splunk index> rm -rf /opt/splunk/etc/auth/distServerKeys/mysearchheadhost

On the search head readd the indexer via the gui.

This will recreate the directory on the indexer and resend the key across to the index and it should fix this issue.

Motivator

Hi Lucas, when you say "On the search head readd the indexer via the gui" do you mean add this indexer as a search peer of that search head again?

0 Karma

Motivator

I deleted the folder $SPLUNK_HOME/etc/auth/distServerKeys/mysearchheadhost. I did not see a message saying that replication had failed for the indexer. I entered the admin/password and saved but the warning is still there. The warning occurs once a minute on each indexer.

0 Karma

Motivator

Mysearchheadhost should be replaced with the name of your actual search head!!! There should be a few directories in there so have a look first (don't cut and paste the command as I posted it)

0 Karma

Motivator

mysearchheadhost was replaced with the name of the search head. This is a Windows instance so I did not use your command. I deleted the "mysearchheadhost" directory then I logged in to splunkweb on the search head and re-entered the admin/password details for the indexer in the distributed search peers view. Note, I did not delete the search peer, I just re-entered to admin/password details and "mysearchheadhost" directory re-appeared on the indexer. What is the name of the key that must be deleted? The only key found in "mysearchheadhost" directory is trusted.pem

0 Karma

Motivator

You haven't deleted the right key if you don't get the replication failure. Double check that you deleted the right one off the indexer.

0 Karma

Motivator

You shouldn't need to readd it as it should still be listed on the search head (you removed it from the indexer only remember). All you need to do is re-auth it. You should see that the replication has failed for that particular search peer. Just click on its name and put in the remote admin/password and click save.

This will recreate the directory on the indexer and copy across the public key from the search head.

I did this for 2 indexers today hence me stumbling on this un answered question when I was looking for the solution 🙂

0 Karma

Motivator

OK I will try this when I have a moment and see if it fixes the issue. Thanks

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!