Getting Error "TCP output processor has paused the data flow"

[rahusri2]

Hello Community,

I am trying to create a connection so that I can sent metric running on 8125 port UDP on Splunk Enterprise (running locally) to Spunk Cloud (running prd-p-7mh2z.splunkcloud.com) but I am getting below error.

As I need to send UDP data running on port 8125, I am using heavy forwarder instead of universal forwarder and I have configured heavy forwarder pointing to "prd-p-7mh2z.splunkcloud.com:9997"

• Getting error on the dashboard

```

The TCP output processor has paused the data flow. Forwarding to host_dest=prd-p-7mh2z.splunkcloud.com inside output group default-autolb-group from host_src=rahusri2s-MacBook-Pro.local has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

```

cat /Applications/splunk/etc/system/local/outputs.conf
Password:
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 1

[tcpout:default-autolb-group]
server = prd-p-7mh2z.splunkcloud.com:9997

[tcpout-server://prd-p-7mh2z.splunkcloud.com:9997]

# cat /Applications/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip

[udp://8125]
connection_host = dns
host = rahusri2s-MacBook-Pro.local
index = 4_dec_8125_udp
sourcetype = statsd

Thanks in advance.

#splunk 

[gcusello]

Hi @rahusri2 ,

let m,e understand: you have a Forwarder (UF or HF) using the outputs.conf you shared to forward logs to Splunk C loud that receives syslogs (using UDP on port 8125), is it correct?

At first my hint is to not using Splunk as receiver but an rsyslog (or syslog-ng or SC4S) to receive syslogs because in this way, you can continue to receive syslogs even if Splunk is down.

Then you can us an UF or an HF to read these files and forward them to Splunk Cloud.ù

In addition, you could have at least two (or more) UFs to receive syslogs with a Load Balancer in front to have a real HA and don't lose data.

But the error you have is probably another one: to send logs to Splunk Cloud from a Forwarder, you have to download an app (called forwarder) from your Splunk Cloud instance, containing the certificates and the passwords to conne ct to Splunk Cloud, you cannot send logs without it.

for more infos see at https://docs.splunk.com/Documentation/Splunk/9.3.2/Data/UsingforwardingagentsCloud

Ciao.

Giuseppe

[rahusri2]

Hello @gcusello,

Thanks for your reply, really appreciated.

---

let m,e understand: you have a Forwarder (UF or HF) using the outputs.conf you shared to forward logs to Splunk C loud that receives syslogs (using UDP on port 8125), is it correct?

I have a StatsD server configured on my local, running on port 8125 (UDP), and it generates some metric data. Currently, this application using statsd server is sending metrics to Splunk Enterprise (running locally). I can view all the metrics from the Splunk analytics workspace without any issues.

Now, I want to forward all application metrics from the StatsD server (running on port 8125 UDP) to Splunk Cloud instead Splunk Enterprise.

I have read in couple of document, for this use case we have to use heavy fordwarder. To achieve this, I added the Splunk Cloud address "prd-p-7mh2z.splunkcloud.com:9997" in "Forwarding and receiving → Configure forwarding" but encountering the following error:

The TCP output processor has paused the data flow. Forwarding to host_dest=prd-p-7mh2z.splunkcloud.com inside output group default-autolb-group from host_src=rahusri2s-MacBook-Pro.local has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

# cat /Applications/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 1

[tcpout:default-autolb-group]
server = prd-p-7mh2z.splunkcloud.com:9997

# cat /Applications/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip

[udp://8125]
connection_host = dns
host = rahusri2s-MacBook-Pro.local
index = 4_dec_8125_udp
sourcetype = statsd

Thank You.

[isoutamo]

Hi

are you sure that this is the correct outputs.conf definition for your host to sending events into SCP?

Usually this is named something like  100_<your splunk stack name>.

You can check the real configurations by 

splunk btool outputs list tcpout --debug

This shows what those configurations are and where those are defined.

Basically you should use that UF configuration which you have downloaded from your SCP stack.

r. Ismo 

[gcusello]

Hi @rahusri2 ,

as I said,you can install a Splunk Heavy Forwarder and configure it exactly as the on-premise receiver.

Then, to forward data to Splunk Cloud, you have to download from your Splunk Cloud instance the Forwarders app and install it on the Heavy Forwarder, otherwise it cannot send logs to Splunk Cloud.

Ciao.

Giuseppe