Hello all,
we built a new cluster as we are getting out of space on current one and we are trying to reroute some of the ingestion to the new cluster by adding the new indexer clusters stanza in the outputs.conf and using _TCP_ROUTING setting in the inputs.conf the servers we want to reroute the ingestion.
below is the stanza we added in outputs.conf
[tcpout:ABC_indexers]
Server = xx.xx.xx.xx.xx:9997, xx.xx.xx.xx.xx:9997, xx.xx.xx.xx:9997
useACK = true
in the inputs.conf we added below setting and pushed it to the servers we want to reroute the data and restarted the forwarder service:
_TCP_ROUTING = ABC_indexers
but we are not seeing any ingestion to the new cluster and we are getting few errors and warning. We checked that the forwarders are connected to all our new indexers over 9997 port.
WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group ABC_indexers has been blocked for 800 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
"INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled."
we checked everything on the indexers but could not find out what is blocking the indexers to receive the data. We have cluster master which is ingesting internal logs to this new indexers and that is not having any issue.
Please let me know if anyone got this issue and how you resolved it.
Thanks
