We built a new indexer cluster and trying to reroute some of the ingestion from current cluster to new cluster

sathwik067 — Wed, 09 Dec 2020 17:50:04 GMT

Hello all,

we built a new cluster as we are getting out of space on current one and we are trying to reroute some of the ingestion to the new cluster by adding the new indexer clusters stanza in the outputs.conf and using _TCP_ROUTING setting in the inputs.conf the servers we want to reroute the ingestion.

below is the stanza we added in outputs.conf

[tcpout:ABC_indexers]

Server = xx.xx.xx.xx.xx:9997, xx.xx.xx.xx.xx:9997, xx.xx.xx.xx:9997

useACK = true

in the inputs.conf we added below setting and pushed it to the servers we want to reroute the data and restarted the forwarder service:

_TCP_ROUTING = ABC_indexers

but we are not seeing any ingestion to the new cluster and we are getting few errors and warning. We checked that the forwarders are connected to all our new indexers over 9997 port.

WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group ABC_indexers has been blocked for 800 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
"INFO ProxyConfig - Failed to initialize https_proxy from server.conf for splunkd. Please make sure that the https_proxy property is set as https_proxy=http://host:port in case HTTP proxying needs to be enabled."

we checked everything on the indexers but could not find out what is blocking the indexers to receive the data. We have cluster master which is ingesting internal logs to this new indexers and that is not having any issue.

Please let me know if anyone got this issue and how you resolved it.

Thanks

Re: We built a new indexer cluster and trying to reroute some of the ingestion from current cluster to new cluster

somesoni2 — Wed, 09 Dec 2020 18:16:26 GMT

Check if the new indexers have receiver enabled correctly: https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Enableareceiver

See if you could send some dummy non-internal data from cluster master (using add oneshot method OR HEC).

topic Re: We built a new indexer cluster and trying to reroute some of the ingestion from current cluster to new cluster in Splunk Enterprise

We built a new indexer cluster and trying to reroute some of the ingestion from current cluster to new cluster

Re: We built a new indexer cluster and trying to reroute some of the ingestion from current cluster to new cluster