Hello all,
I am at a bit loss in what to do at this point. I had an indexer fail and now that my it is healthy I cannot rejoin the cluster reliably. After a reboot/restarting splunk he will join and begin syncing buckets for about 10 mins. After this he will throw errors and get stuck in a batch adding state on the indexer management page. I get this error on the master:
Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=502 expected_response_code=2xx status_line="Bad request" socket_error="No error" remote_error= [ event=ReaddPeer status=retrying AddPeerRequest: { _id= active_bundle_id=9884CA425F0224F22F37BE784337C463 add_type=ReAdd base_generation_id=0 batch_serialno=1 batch_size=4 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=9884CA425F0224F22F37BE784337C463 mgmt_port=8089 name=6A1C1358-0C02-4D60-B58B-EA903E3D0991 register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name= site=default splunk_version=8.0.7 splunkd_build_number=1c4f3bbe1aea status=Up } ].
The indexer in question displays the same error, along with this one. I’m not sure if it’s related.
ERROR HTTPClientRequest: caught exception while parsing HTTP Reply: string value too long value size = 531110, Maxvaluesize = 524288
i should mention I did reinstall splunk over the currently installed version as part of fixing that indexer.
