How to fix "Forwarding to output group splunkcloud has been blocked for XXX seconds" errors?

[RodneyNelsonCNP]

I have to Windows Servers (one 2012 and one 2016) that we are unable to receive any events for in our splunkcloud instance.

The errors that keep repeating in the splunkd log are:

ERROR TcpOutputFd - Connection to host=< splunkcloud ip address>:9997 failed. sock_error = 10054. SSL Error = No error

and

ERROR TcpOutputProc - Applying quarantine to ip==< splunkcloud ip address> port=9997 _numberOfFailures=2

WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunkcloud has been blocked for 300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

I've tried the basics like uninstalling splunk reinstalling it.
I am able to telent from either of these servers to the specified splunk ID via port 9997 without an issue.

I've searched this site and find people who have had similar issues but they were all for local instances of splunk enterprise and reference checking things on the indexer server. Since we are using splunkcloud we do not have the ability to check the indexers.

Please advise...

[RodneyNelsonCNP]

Just an update, I was able to resolve this issue. It turned out to be a janky network fw rule that was blocking certain traffic. Once our network admins removed the janky rule we ere able to successfully send logs to splunk.

We were able to identify this was a FW issue once we noticed that we couldn't get to any single sign on sites (gmail, okta, aws console etc..) from the problematic servers.

[woodcock]

This is a long shot but if this server is very busy with files/ports, you may have run out of file descriptors. If linux, you can check with ulimits.

[lakshman239]

As you have indicated connectivity, I assume there is no n/w connectivity issue. If this windows server is connected directly to the splunk cloud, are there any other windows/linux forwarders successfully sending data to cloud? Can you check limits.conf on the windows server ? Are you sending a lot of old events? [ you can also use 'current_old' OR 'start_from' to limit the data and test that the integration works]
https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Inputsconf

If all fails, raise a support case with splunk cloud support to look at indexers health and see which pipelines are having issues.

[RodneyNelsonCNP]

@lakshman239 yes we have 300+ windows servers that are successfully able to send data to our splunk cloud instance. We also use a splunk deployment server hosted internally that we use to deploy the configuration of the inputs.conf file to make sure all the servers are the same.

The server hasn't been able to send events in a couple of months so I did wonder if maybe it was trying to send to many back logged events. So I did edit the inputs.conf file to set the currently_only = 1 (instead of 0) for the stanza's and then restarted splunk but that didn't seem to have an impact. Splunk Cloud is still not receiving events and the same error appears in the log.

I have asked our Splunk Admin to open a case with support. All of our servers use the same config as far as limits.conf, outputs.conf, inputs.conf files

[markusspitzli]

It might be possible that something is odd with the indexing or parsing queue on the Splunk Cloud Instance. And the problem on your Windows Servers are just sideeffects. But this would have effects on other servers too.

Have a look at the Management Console -> Indexing - Indexing Performance: Deployment.

At the Indexing Performance by Instance Panel you can see if you have issues with any of the queues.

[RodneyNelsonCNP]

Thanks @markusspitzli ... hmm since this is a splunk cloud instance I don't believe I have access to the Management Console. I do not see it listed when I login to splunk cloud. I will have to follow up with our Splunk Admin.

[deepashri_123]

Hey@RodneyNelsonCNP,

Can you please check if you have enabled receiving on 9997?

[RodneyNelsonCNP]

@deepashri_123 yes I'm able to telnet to the splunk cloud ip's via port 9997. We do not have any local FW's on this server turned on.