topic TcpOutputProc Connection to ...... closed. Connection closed by server in Getting Data In

TcpOutputProc Connection to ...... closed. Connection closed by server

foewar — Wed, 03 Jul 2013 11:52:41 GMT

Windows The Log

07-03-2013 17:03:44.654 +0530 WARN  TcpOutputProc - Applying quarantine to ip=107.20.29.58 port=9997 _numberOfFailures=7
07-03-2013 17:03:46.107 +0530 INFO  TcpOutputProc - Connected to idx=50.17.56.245:9997 using ACK.
07-03-2013 17:04:16.112 +0530 INFO  TcpOutputProc - Connected to idx=54.243.3.115:9997 using ACK.
07-03-2013 17:04:46.113 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.
07-03-2013 17:05:14.114 +0530 INFO  TcpOutputProc - Removing quarantine from idx=23.20.94.208:9997
07-03-2013 17:05:14.801 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:14.847 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:14.861 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:15.071 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:15.071 +0530 WARN  TcpOutputProc - Applying quarantine to ip=23.20.94.208 port=9997 _numberOfFailures=18
07-03-2013 17:05:15.109 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:15.111 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:16.072 +0530 INFO  TcpOutputProc - Connected to idx=50.17.56.245:9997 using ACK.
07-03-2013 17:05:44.082 +0530 INFO  TcpOutputProc - Removing quarantine from idx=107.20.29.58:9997
07-03-2013 17:05:47.082 +0530 INFO  TcpOutputProc - Connected to idx=107.22.10.147:9997 using ACK.
07-03-2013 17:06:16.083 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.
07-03-2013 17:06:45.957 +0530 INFO  TcpOutputProc - Connected to idx=107.22.10.147:9997 using ACK.
07-03-2013 17:07:15.962 +0530 INFO  TcpOutputProc - Connected to idx=54.243.3.115:9997 using ACK.
07-03-2013 17:07:45.963 +0530 INFO  TcpOutputProc - Connected to idx=23.22.208.232:9997 using ACK.
07-03-2013 17:08:16.059 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.

Not sure what is causing this ..

/opt/splunkforwarder/etc/system/default/outputs.conf

#   Version 5.0.3

[tcpout]
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
useACK = false

 /opt/splunkforwarder/etc/system/default/inputs.conf

#   Version 5.0.3
# DO NOT EDIT THIS FILE!
# Please make all changes to files in $SPLUNK_HOME/etc/system/local.
# To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/system/default
# into ../local and edit there.
#
# This file contains possible attributes and values you can use to
# configure inputs, distributed inputs and file system monitoring.


[default]
index         = default
_rcvbuf        = 1572864
host = $decideOnStartup
evt_resolve_ad_obj = 0
evt_dc_name=
evt_dns_name=



[monitor://$SPLUNK_HOME\var\log\splunk]
index = _internal

[monitor://$SPLUNK_HOME\etc\splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME\var\spool\splunk]
move_policy = sinkhole
crcSalt = <SOURCE>

[batch://$SPLUNK_HOME\var\spool\splunk\...stash_new]
queue       = stashparsing
sourcetype  = stash_new
move_policy = sinkhole
crcSalt     = <SOURCE>


[fschange:$SPLUNK_HOME\etc]
#poll every 10 minutes
pollPeriod = 600
#generate audit events into the audit index, instead of fschange events
signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0

[SSL]
# default cipher suites that splunk allows. Change this if you wish to increase the security 
# of SSL connections, or to lower it if you having trouble connecting to splunk.
cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM


[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
interval = 10000000
source = wmi
sourcetype = wmi
queue = winparsing
persistentQueueSize=200MB

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0
interval = 10000000
source = WinRegistry 
sourcetype=WinRegistry
queue = winparsing
persistentQueueSize=50MB

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
interval = 10000000
source = ActiveDirectory
sourcetype = ActiveDirectory
disabled = 0
queue = winparsing
persistentQueueSize=50MB

[WinEventLog:Application]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:Security]
disabled = 1
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5

[WinEventLog:Setup]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:System]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:ForwardedEvents]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:HardwareEvents]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:Internet Explorer]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5


Any help would be appreciated

Re: TcpOutputProc Connection to ...... closed. Connection closed by server

phoffman_splunk — Wed, 03 Jul 2013 13:45:21 GMT

You should check in your indexers splunkd.log and metrics.log to see what is going on with your indexer. Can't tell much from looking at one side of the issue.

If you still don't get enough info you can crank up the logging on both sides to get more detail.

method of increasing logging on inputs and outputs: http://blogs.splunk.com/2008/09/22/enabling-debug-messages/

Re: TcpOutputProc Connection to ...... closed. Connection closed by server

imagineer411 — Tue, 26 Nov 2013 23:18:25 GMT

I've encountered similar issue with slightly different logs as follows:

11-26-2013 23:03:58.619 +0000 WARN TcpOutputProc - Shutdown timed out for 107.22.10.147:9997
11-26-2013 23:04:28.607 +0000 WARN TcpOutputProc - Raw connection to ip=107.22.10.147:9997 timed out
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Detected connection to 107.22.10.147:9997 closed
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Will close stream to current indexer 107.22.10.147:9997
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Closing stream for idx=107.22.10.147:9997
11-26-2013 23:04:58.929 +0000 WARN TcpOutputProc - Cooked connection to ip=107.20.29.58:9997 timed out
11-26-2013 23:05:18.931 +0000 WARN TcpOutputProc - Cooked connection to ip=23.23.14.246:9997 timed out
11-26-2013 23:05:38.931 +0000 WARN TcpOutputProc - Cooked connection to ip=107.22.148.176:9997 timed out
11-26-2013 23:05:58.933 +0000 WARN TcpOutputProc - Cooked connection to ip=54.243.3.115:9997 timed out
11-26-2013 23:06:18.936 +0000 WARN TcpOutputProc - Cooked connection to ip=50.17.56.245:9997 timed out
11-26-2013 23:06:20.068 +0000 INFO TcpOutputProc - Connected to idx=54.224.46.188:9997 using ACK.
11-26-2013 23:06:58.937 +0000 WARN TcpOutputProc - Cooked connection to ip=23.22.208.232:9997 timed out
11-26-2013 23:07:18.940 +0000 WARN TcpOutputProc - Cooked connection to ip=107.20.29.58:9997 timed out
11-26-2013 23:07:20.942 +0000 INFO TcpOutputProc - Connected to idx=23.20.94.208:9997 using ACK.
11-26-2013 23:07:28.600 +0000 WARN TcpOutputProc - Shutdown timed out for 23.23.14.246:9997
11-26-2013 23:07:40.590 +0000 INFO TcpOutputProc - Connected to idx=54.224.135.13:9997 using ACK.

If I'm using splunk>storm then I won't be able to see indexers splunkd.log and metrics.log to see what is going on with the indexer, right? If so, where do I go from here?

My setup is as follows:
webserver nodes->splunkforwarder->splunkstorm

Both webserver nodes and splunkforwarder shows active forwards.

Any help is greatly appreciated.
Thank you.

Re: TcpOutputProc Connection to ...... closed. Connection closed by server

lukejadamec — Wed, 27 Nov 2013 01:31:56 GMT

When you search for data from this host, are you getting data from this host?

Re: TcpOutputProc Connection to ...... closed. Connection closed by server

phoffman_splunk — Mon, 28 Sep 2020 15:22:32 GMT

You can search the _internal index to see your splunkd and metrics logs. If you want anyone in particular,... index=_internal source=*metrics.log

Re: TcpOutputProc Connection to ...... closed. Connection closed by server

Akili — Tue, 24 Dec 2013 10:20:44 GMT

had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path /SplunkUniversalForwarder/etc/system/local
replace the config files under with those from:
path /SplunkUniversalForwarder/ etc/ apps/Windows /local
restart splunkforwarder:
splunk restart

it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.