Version:
Splunk Enterprise 7.2.9.1
Problem:
If I add a Forwarder, in this case a Heavy Forwarder -- all data flow to/from all Forwarders stops.
This occurs when I am adding the forwarder using the Splunk web interface (Settings >> Forwarding and Receiving >> "New Forwarding Host") and/or using the command prompt.
I start receiving messages citing:
"The TCP output processor has paused the data flow. Forwarding to host_dest=###### from host_src=###### has been blocked for blocked_seconds=####"...."This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data."
And after 15 of so minutes, all Forwarder statuses are Missing.
Splunk remains in this state indefinitely until I remove the just added Forwarder(s).
Intention
Turn a Universal Forwarder into a Heavy Forwarder: to route Indexed records from this child domain's Indexer to the Enterprise-level domain Splunk server.
Are you using this document to configure the forward has HF?
https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Deployaheavyforwarder
Could you provide the output of "Splunk btool output list --debug" on the configured HF? Are you just configure Splunk servers like indexer to the HF with output.conf. Are you using an SSL connection from the sub-domain to the primary domain indexer?
Please share the command you are using to add a heavy forwarder and where you are running it. Usually, all that is required is to add the IP addresses of the indexers to the forwarder's outputs.conf file (not counting any firewalls, of course).
It is not possible to turn a universal forwarder into a heavy forwarder. They are two different software installations.