Security & the Enterprise
Much secured. So patch!

Adding Heavy Forwarder disrupts data flow to/from all Forwarders

SyntaxError
Engager

Version:

Splunk Enterprise 7.2.9.1

 

Problem:

If I add a Forwarder, in this case a Heavy Forwarder -- all data flow to/from all Forwarders stops.

This occurs when I am adding the forwarder using the Splunk web interface (Settings >> Forwarding and Receiving >> "New Forwarding Host") and/or using the command prompt.

I start receiving messages citing:

"The TCP output processor has paused the data flow. Forwarding to host_dest=###### from host_src=###### has been blocked for blocked_seconds=####"...."This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data."

And after 15 of so minutes, all Forwarder statuses are Missing.

Splunk remains in this state indefinitely until I remove the just added Forwarder(s).

 

Intention

Turn a Universal Forwarder into a Heavy Forwarder:  to route Indexed records from this child domain's Indexer to the Enterprise-level domain Splunk server.

 

0 Karma

youngsuh
Contributor

Are you using this document to configure the forward has HF?

https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Deployaheavyforwarder

Could you provide the output of "Splunk btool output list --debug" on the configured HF?  Are you just configure Splunk servers like indexer to the HF with output.conf.  Are you using an SSL connection from the sub-domain to the primary domain indexer?

richgalloway
SplunkTrust
SplunkTrust

Please share the command you are using to add a heavy forwarder and where you are running it.  Usually, all that is required is to add the IP addresses of the indexers to the forwarder's outputs.conf file (not counting any firewalls, of course).

It is not possible to turn a universal forwarder into a heavy forwarder.  They are two different software installations.

---
If this reply helps you, Karma would be appreciated.

SyntaxError
Engager
Thanks for the insight.  Now, I know that I'm not approaching this in the best way.  But I don't even know if what I'm doing is correct.  Let me re-write this.  
 
I am trying to send the Indexer logs and records to a Forwarder (both on a separate, child domain).  That Forwarder also interfaces with the Enterprise domain on a secondary NIC.  I need to configure that Forwarder to receive (request?) the Indexer logs and records (from child domain) and forward those to the Enterprise Splunk server (on the main domain).
 
Is this possible with a Universal Forwarder?  Or it is, but only with a Heavy Forwarder? 
Your help is appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...