Splunk Cloud: splunkd.log -The TCP output processor has paused the data flow.

[sreek]

Hi,

I am attempting to send syslog data from WAF to a Heavy Forwarder (HF) over port 9515, and then forward it to Splunk Cloud. From the tcpdump analysis, I can confirm that the data is being received by the HF. However, it seems that the HF is not forwarding the data to Splunk Cloud.

inputs

[tcp://9515]
disabled = false
connection_host=ip
sourcetype = f5:bigip:syslog

I have already set up the necessary inputs in the HF to receive syslog data via TCP port 9515 and configured the outputs using the Splunk Cloud Forwarder Credential app.

In the logs, I have observed the following errors:

tail -f /opt/splunk/var/log/splunk/splunkd.log

WARN TcpOutputProc [154415 indexerPipe] - The TCP output processor has paused the data flow. Forwarding to host_dest=<Splunk_Cloud_Indexer> inside output group splunkcloud_outgroup from host_src=<heavy_forwarder_ip> has been blocked for blocked_seconds=60. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

WARN AutoLoadBalancedConnectionStrategy [155227 TcpOutEloop] - Cooked connection to ip=<Splunk_Cloud_Indexer>:9997 timed out

ERROR DispatchManager [147593 TcpChannelThread] - The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=splunk-user

[isoutamo]

Hi

there could be three reason why it said that.

1. your splunk cloud is out of disk space, which sounds really weird. Fix: ask splunk cloud support and get they fix this if option 2 is not a reason.
   
   
2. You haven’t configure HF just forwarding events to SC. It’s also indexing those and now it’s lack of disk space. Fix. Configure your HF just forwarding not also indexing events and free needed space to get it working again. Of course it could l be that it’s already configured correctly, but its /opt/splunk is too small to store all needed logs etc. if this is true you must increase that file system.
3. You are collecting that data to disk and don’t just forward it to Splunk cloud. Fix: Don’t collect it to disk, just send feed to splunk cloud.

r. Ismo