If I add a Forwarder, in this case a Heavy Forwarder -- all data flow to/from all Forwarders stops.
This occurs when I am adding the forwarder using the Splunk web interface (Settings >> Forwarding and Receiving >> "New Forwarding Host") and/or using the command prompt.
I start receiving messages citing:
"The TCP output processor has paused the data flow. Forwarding to host_dest=###### from host_src=###### has been blocked for blocked_seconds=####"...."This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data."
And after 15 of so minutes, all Forwarder statuses are Missing.
Splunk remains in this state indefinitely until I remove the just added Forwarder(s).
Turn a Universal Forwarder into a Heavy Forwarder: to route Indexed records from this child domain's Indexer to the Enterprise-level domain Splunk server.
Could you provide the output of "Splunk btool output list --debug" on the configured HF? Are you just configure Splunk servers like indexer to the HF with output.conf. Are you using an SSL connection from the sub-domain to the primary domain indexer?
Please share the command you are using to add a heavy forwarder and where you are running it. Usually, all that is required is to add the IP addresses of the indexers to the forwarder's outputs.conf file (not counting any firewalls, of course).
It is not possible to turn a universal forwarder into a heavy forwarder. They are two different software installations.
--- If this reply helps you, an upvote would be appreciated.
Thanks for the insight. Now, I know that I'm not approaching this in the best way. But I don't even know if what I'm doing is correct. Let me re-write this.
I am trying to send the Indexer logs and records to a Forwarder (both on a separate, child domain). That Forwarder also interfaces with the Enterprise domain on a secondary NIC. I need to configure that Forwarder to receive (request?) the Indexer logs and records (from child domain) and forward those to the Enterprise Splunk server (on the main domain).
Is this possible with a Universal Forwarder? Or it is, but only with a Heavy Forwarder?