Getting Data In

Why has The TCP output processor paused data flow?

carlyleadmin
Contributor

Hi,

i am not able to receive any data from my forwarder. It stopped working yesterday.port 9997 is open.connection is established.i can telnet to my server(which is my laptop).

here is the error from the splunkd from the forwarder

09-21-2017 15:20:51.293 -0400 WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 82200 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

here is my input file from the forwarder server
[default]
host = HC1xxxxxxCV

[monitor://C:\Program Files (x86)..........]
disabled = false
followTail = 0
sourcetype=Data Import
ignoreOlderThan = 6d

here is my outputs file from the forwarder

[tcpout]
defaultGroup = default-autolb-group

[tcpout-server://rs1-sbaba-t440.xxxxxxxx:9997]

[tcpout:default-autolb-group]
disabled = false
server = rs1-sbaba-t440.xxxxxxxxxxxxx:9997,rs1-sbaba-t440:9997

[tcpout-server://rs1-sbaba-t440:9997]

i checked on my laptop the reeving is enabled

this is my input file from the receiver

[default]
host = rs1-sbaba-t440

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

i read something about internal que being blocked.and set the stopacceptafterqblock attribute on inputs file which i dont see in my receivers inputs file(under local folder)never change the conf files under default folder.

i've been banging my head for hours and since i am stuck probably missing something very simple but can't find it.anyhelp is appreciated.

this thing was working fine till yesterday and now all of a sudden i am not bale to get data

thanks,

0 Karma
1 Solution

carlyleadmin
Contributor
0 Karma

sbbadri
Motivator

Try this

1) check indexers have enough space.
2) check Licesnse should not cross daily limit.
3) Try to simply outputs.conf like below.
outputs.conf.

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
disabled = false
server = rs1-sbaba-t440.xxxxxxxxxxxxx:9997,rs1-sbaba-t440:9997

0 Karma

carlyleadmin
Contributor

i just uninstalled the forwarder and reinstall it and it started working agan.it is odd

0 Karma

carlyleadmin
Contributor
0 Karma

thejohn
Path Finder

Hi, i have the same problem but i don't understand how to resolve it.
Can you explicit me the solutions?
THanks

0 Karma

jkat54
SplunkTrust
SplunkTrust

You should have something like

[splunktcp://9997]

In inputs.conf on the indexer. Which in this case is your laptop it sounds like.

Probably easiest for you to open the Splunk web ui on your laptop log in as admin, go to settings -> forwarding and receiving -> receiving ... enable receiving and specify port 9997.

0 Karma

carlyleadmin
Contributor

receiving is enabled thru gui but when i check the inputs.config i dont see the stanza.

i tried adding it manually to the config and still no good.it shows inactive on forwarder when search for it.

i am on a trial license would that be the reason.daily volume issue?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Something isn't configured right... network, firewall, Splunk, etc.

On the forwarder, run this command:

$SPLUNK_HOME/bin/splunk btool outputs list --debug

On the indexer, run this command:

$SPLUNK_HOME/bin/splunk btool inputs list --debug

Make sure you're replacing $SPLUNK_HOME with the actual path to the Splunk folder.

Copy and paste the outputs here. Then we should be able to tell you if Splunk is configured correctly.

0 Karma

thejohn
Path Finder

I show you my outputs list, can you check if it's all correct? Thanks

/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/local/outputs.conf indexAndForward = 0
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://192.168.5.42:9997]
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/system/local/outputs.conf disabled = 0
/opt/splunkforwarder/etc/system/local/outputs.conf server = 192.168.5.42:9997

0 Karma

jkat54
SplunkTrust
SplunkTrust

Open a new thread please.

0 Karma