Why is Indexer Discovery on a Splunk 6.3.3 universal forwarder failing with http_response=Unauthorized?

[joshuabiggley]

We followed the documentation as specified, but when we configure the universal forwarders as specified we get the error below

02-26-2016 08:47:11.754 -0600 ERROR HttpClientRequest - Caught exception while parsing HTTP reply: Unexpected character while looking for value: '<'
02-26-2016 08:47:11.754 -0600 ERROR IndexerDiscoveryHeartbeatThread - failed heartbeat for group=splunkssl uri=https://X.X.X.X:8089/services/indexer_discovery http_response=Unauthorized

I've seen this error posted in the forums elsewhere without resolution.

We're running Splunk 6.3.3

Any thoughts?

[joshuabiggley · Accepted solution]

Although we thought were were careful enough, we changed the /opt/splunk/etc/system/local/server.conf file so that the [general] and [indexer_discovery] stanzas both had the same pass4SymmKey. We then updated the outputs.conf on the universal forwarders in question and restarted the universal forwarder service and it worked.

I could have sworn that we followed that process previously, but we obviously borked something the first time through.

Marking as resolved.