Hello all,
I am at a bit loss in what to do at this point. I had an indexer fail and now that my it is healthy I cannot rejoin the cluster reliably. After a reboot/restarting splunk he will join and begin syncing buckets for about 10 mins. After this he will throw errors and get stuck in a batch adding state on the indexer management page. I get this error on the master:
Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=502 expected_response_code=2xx status_line="Bad request" socket_error="No error" remote_error= [ event=ReaddPeer status=retrying AddPeerRequest: { _id= active_bundle_id=9884CA425F0224F22F37BE784337C463 add_type=ReAdd base_generation_id=0 batch_serialno=1 batch_size=4 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=9884CA425F0224F22F37BE784337C463 mgmt_port=8089 name=6A1C1358-0C02-4D60-B58B-EA903E3D0991 register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name= site=default splunk_version=8.0.7 splunkd_build_number=1c4f3bbe1aea status=Up } ].
The indexer in question displays the same error, along with this one. I’m not sure if it’s related.
ERROR HTTPClientRequest: caught exception while parsing HTTP Reply: string value too long value size = 531110, Maxvaluesize = 524288
i should mention I did reinstall splunk over the currently installed version as part of fixing that indexer.
I was able to fix this issue by rebuilding the server, there was some corruption in the filesystem.
I was able to fix this issue by rebuilding the server, there was some corruption in the filesystem.
After your reinstalled Splunk did you update the pass4SymmKey's in server.conf under [general] and [clustering]? Those have to match in order to join the cluster. Also, you'll need to enter the plain text version, don't copy the hashed pass4SymmKey from another node.
Thank you for the suggestion, I did verify this but it ended up being a different issue that I was able to resolve.