We have index clustering working fine. We have several heavy forwarders configured successfully with indexer discovery. However, when I go to add another new forwarder, I get the issue below. My steps are to add the clear text pass phrase in the outputs.conf of the forwarder. Then I restart the forwarder. From there Splunk encrypts to what is below (pass4SymmKey). Is this the correct way to add discovery to a forwarder?
[tcpout:default-autolb-group] indexerDiscovery = cluster useACK = true [indexer_discovery:cluster] master_uri = https://cluster_master:8089 pass4SymmKey = $1$19GA9JbHEqO/13Z8+c4/2Q== 10-04-2016 13:16:50.955 -0400 ERROR IndexerDiscoveryHeartbeatThread - failed heartbeat for group=default-autolb-group uri=https://cluster_master:8089/services/indexer_discovery http_response=Unauthorized 10-04-2016 13:16:52.066 -0400 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 810 seconds.
I believe you have encountered a product defect that is fixed in 6.5.0, 6.4.4 (released last night) and 6.3.7. We have not seen the issue on anything 6.2 and earlier. Documentation is being updated to reflect that this is a fixed issue in 6.5.0.
Upgrading to 6.4.4 should fix your issue
Saw this yesterday and, since our Splunk cluster isn't in production yet, I performed the upgrade to 6.5.0 this morning, but still receive the same results. I've double-checked that the pass4SymmKey values match.
Any possibility that I would need to upgrade the UF as well?