Errors when upgrading from 5.0.4 to 6.1.3.

[immortalraghava]

Hi,

We are upgrading from 5.0.4 to 6.1.3 in a distributed Splunk environment. There are multiple instances of Splunk running as Forwarder, Indexer, and single Deployment server(License Master) which also runs some scheduled searches and sends data back to Indexers. In our upgrade process we stopped Splunk process in the node extract Splunk 6.1.3 on the existing Splunk installation path and again started Splunk with "Splunk start accept license" command. We do this node by node starting with Indexer, forwarder, and deployment server. Now after installation is complete in all nodes, the deployment server splunkd.log contains the following error messages

GetRemoteAuthToken - Unable to get authentication token from peeruri="https://INDEXER:8089/services/admin/auth-tokens

Unable to distribute to peer named INDEXER at uri https://<Indexer>:8089 because peer has status = "Authentication Failed"

The following procedure resolves this error
Remove the indexer entry from distsearch.conf file and restart Splunk.

[distributedSearch]
servers = INDEXER:8089

After that again add the same entry in the same file and restart Splunk.

This happens only after the upgrade.
Any help is appreciated.
Thanks!

[davebo1896]

I stumbled across this when upgrading to 6.3.0
I think it is related to the distServerKeys being regenerated.
Make sure to save your private.pem and trusted.pem as well as the remote keys from other servers and put them back into etc/auth/distServerKeys/.

[jkat54]

each indexer has a $SPLUNK_HOME/etc/auth/distServerKeys/{SEARCH_HEAD}/trusted.pem for each search head and the cluster master.

If for whatever reason your indexer "knew" of your search head in the past, and you've recently rebuilt your search head or created a clone with the same servername, etc. then you may have to delete the {SEARCH_HEAD} dir show above on each indexer that previously knew of the search head. Then, add the peer again via the UI on the search head.

It happened to us when we changed from single search head to search head cluster, and we re-used the old search head in the new SHC.

[lakshman237]

Hopefully this resolved by now. This usually appears when the search head cannot authenticate with search peers in the distributed environment. Pls re-authenticate either using CLI or from the GUI within the search head : settings-> distributed search -> Search peers and ensure, its up and enabled.