Why has The TCP output processor paused data flow?

[carlyleadmin]

Hi,

i am not able to receive any data from my forwarder. It stopped working yesterday.port 9997 is open.connection is established.i can telnet to my server(which is my laptop).

here is the error from the splunkd from the forwarder

09-21-2017 15:20:51.293 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 82200 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

here is my input file from the forwarder server
[default]
host = HC1xxxxxxCV

[monitor://C:\Program Files (x86)..........]
disabled = false
followTail = 0
sourcetype=Data Import
ignoreOlderThan = 6d

here is my outputs file from the forwarder

[tcpout]
defaultGroup = default-autolb-group

[tcpout-server://rs1-sbaba-t440.xxxxxxxx:9997]

[tcpout:default-autolb-group]
disabled = false
server = rs1-sbaba-t440.xxxxxxxxxxxxx:9997,rs1-sbaba-t440:9997

[tcpout-server://rs1-sbaba-t440:9997]

i checked on my laptop the reeving is enabled

this is my input file from the receiver

[default]
host = rs1-sbaba-t440

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

i read something about internal que being blocked.and set the stopacceptafterqblock attribute on inputs file which i dont see in my receivers inputs file(under local folder)never change the conf files under default folder.

i've been banging my head for hours and since i am stuck probably missing something very simple but can't find it.anyhelp is appreciated.

this thing was working fine till yesterday and now all of a sudden i am not bale to get data

thanks,

[MattHatter]

In my case, I forgot to enable listening on my indexers. (You'll need to do this on any heavy forwarders forwarders as well). 

/opt/splunk/bin/splunk enable listen 9997

[sbbadri]

Try this

1) check indexers have enough space.
2) check Licesnse should not cross daily limit.
3) Try to simply outputs.conf like below.
outputs.conf.

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = rs1-sbaba-t440.xxxxxxxxxxxxx:9997,rs1-sbaba-t440:9997

[carlyleadmin]

i just uninstalled the forwarder and reinstall it and it started working agan.it is odd

[carlyleadmin · Accepted solution]

[thejohn]

Hi, i have the same problem but i don't understand how to resolve it.
Can you explicit me the solutions?
THanks

[jkat54]

You should have something like

[splunktcp://9997]

In inputs.conf on the indexer. Which in this case is your laptop it sounds like.

Probably easiest for you to open the Splunk web ui on your laptop log in as admin, go to settings -> forwarding and receiving -> receiving ... enable receiving and specify port 9997.

[carlyleadmin]

receiving is enabled thru gui but when i check the inputs.config i dont see the stanza.

i tried adding it manually to the config and still no good.it shows inactive on forwarder when search for it.

i am on a trial license would that be the reason.daily volume issue?

[jkat54]

Something isn't configured right... network, firewall, Splunk, etc.

On the forwarder, run this command:

$SPLUNK_HOME/bin/splunk btool outputs list --debug

On the indexer, run this command:

$SPLUNK_HOME/bin/splunk btool inputs list --debug

Make sure you're replacing $SPLUNK_HOME with the actual path to the Splunk folder.

Copy and paste the outputs here. Then we should be able to tell you if Splunk is configured correctly.

[thejohn]

I show you my outputs list, can you check if it's all correct? Thanks

/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/local/outputs.conf indexAndForward = 0
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://192.168.5.42:9997]
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/system/local/outputs.conf disabled = 0
/opt/splunkforwarder/etc/system/local/outputs.conf server = 192.168.5.42:9997

[jkat54]

Open a new thread please.