Getting Data In
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

TcpOutputProc Connection to ...... closed. Connection closed by server

foewar
New Member
‎07-03-2013 04:52 AM 
Windows The Log

07-03-2013 17:03:44.654 +0530 WARN  TcpOutputProc - Applying quarantine to ip=107.20.29.58 port=9997 _numberOfFailures=7
07-03-2013 17:03:46.107 +0530 INFO  TcpOutputProc - Connected to idx=50.17.56.245:9997 using ACK.
07-03-2013 17:04:16.112 +0530 INFO  TcpOutputProc - Connected to idx=54.243.3.115:9997 using ACK.
07-03-2013 17:04:46.113 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.
07-03-2013 17:05:14.114 +0530 INFO  TcpOutputProc - Removing quarantine from idx=23.20.94.208:9997
07-03-2013 17:05:14.801 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:14.847 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:14.861 +0530 INFO  TcpOutputProc - Connected to idx=23.20.94.208:9997
07-03-2013 17:05:15.071 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:15.071 +0530 WARN  TcpOutputProc - Applying quarantine to ip=23.20.94.208 port=9997 _numberOfFailures=18
07-03-2013 17:05:15.109 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:15.111 +0530 INFO  TcpOutputProc - Connection to 23.20.94.208:9997 closed. Connection closed by server.
07-03-2013 17:05:16.072 +0530 INFO  TcpOutputProc - Connected to idx=50.17.56.245:9997 using ACK.
07-03-2013 17:05:44.082 +0530 INFO  TcpOutputProc - Removing quarantine from idx=107.20.29.58:9997
07-03-2013 17:05:47.082 +0530 INFO  TcpOutputProc - Connected to idx=107.22.10.147:9997 using ACK.
07-03-2013 17:06:16.083 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.
07-03-2013 17:06:45.957 +0530 INFO  TcpOutputProc - Connected to idx=107.22.10.147:9997 using ACK.
07-03-2013 17:07:15.962 +0530 INFO  TcpOutputProc - Connected to idx=54.243.3.115:9997 using ACK.
07-03-2013 17:07:45.963 +0530 INFO  TcpOutputProc - Connected to idx=23.22.208.232:9997 using ACK.
07-03-2013 17:08:16.059 +0530 INFO  TcpOutputProc - Connected to idx=23.22.132.83:9997 using ACK.

Not sure what is causing this ..

/opt/splunkforwarder/etc/system/default/outputs.conf

#   Version 5.0.3

[tcpout]
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
useACK = false

 /opt/splunkforwarder/etc/system/default/inputs.conf

#   Version 5.0.3
# DO NOT EDIT THIS FILE!
# Please make all changes to files in $SPLUNK_HOME/etc/system/local.
# To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/system/default
# into ../local and edit there.
#
# This file contains possible attributes and values you can use to
# configure inputs, distributed inputs and file system monitoring.


[default]
index         = default
_rcvbuf        = 1572864
host = $decideOnStartup
evt_resolve_ad_obj = 0
evt_dc_name=
evt_dns_name=



[monitor://$SPLUNK_HOME\var\log\splunk]
index = _internal

[monitor://$SPLUNK_HOME\etc\splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME\var\spool\splunk]
move_policy = sinkhole
crcSalt = <SOURCE>

[batch://$SPLUNK_HOME\var\spool\splunk\...stash_new]
queue       = stashparsing
sourcetype  = stash_new
move_policy = sinkhole
crcSalt     = <SOURCE>


[fschange:$SPLUNK_HOME\etc]
#poll every 10 minutes
pollPeriod = 600
#generate audit events into the audit index, instead of fschange events
signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0

[SSL]
# default cipher suites that splunk allows. Change this if you wish to increase the security 
# of SSL connections, or to lower it if you having trouble connecting to splunk.
cipherSuite = ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM


[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
interval = 10000000
source = wmi
sourcetype = wmi
queue = winparsing
persistentQueueSize=200MB

[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0
interval = 10000000
source = WinRegistry 
sourcetype=WinRegistry
queue = winparsing
persistentQueueSize=50MB

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
interval = 10000000
source = ActiveDirectory
sourcetype = ActiveDirectory
disabled = 0
queue = winparsing
persistentQueueSize=50MB

[WinEventLog:Application]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:Security]
disabled = 1
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5

[WinEventLog:Setup]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:System]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:ForwardedEvents]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:HardwareEvents]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5

[WinEventLog:Internet Explorer]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5


Any help would be appreciated
0 Karma
Reply

Akili
Path Finder
‎12-24-2013 02:20 AM

had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path /SplunkUniversalForwarder/etc/system/local
replace the config files under with those from:
path /SplunkUniversalForwarder/ etc/ apps/Windows /local
restart splunkforwarder:
splunk restart

it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.

0 Karma
Reply

lukejadamec
Super Champion
‎11-26-2013 05:31 PM

When you search for data from this host, are you getting data from this host?

0 Karma
Reply

imagineer411
Engager
‎11-26-2013 03:18 PM

I've encountered similar issue with slightly different logs as follows:

11-26-2013 23:03:58.619 +0000 WARN TcpOutputProc - Shutdown timed out for 107.22.10.147:9997
11-26-2013 23:04:28.607 +0000 WARN TcpOutputProc - Raw connection to ip=107.22.10.147:9997 timed out
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Detected connection to 107.22.10.147:9997 closed
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Will close stream to current indexer 107.22.10.147:9997
11-26-2013 23:04:28.607 +0000 INFO TcpOutputProc - Closing stream for idx=107.22.10.147:9997
11-26-2013 23:04:58.929 +0000 WARN TcpOutputProc - Cooked connection to ip=107.20.29.58:9997 timed out
11-26-2013 23:05:18.931 +0000 WARN TcpOutputProc - Cooked connection to ip=23.23.14.246:9997 timed out
11-26-2013 23:05:38.931 +0000 WARN TcpOutputProc - Cooked connection to ip=107.22.148.176:9997 timed out
11-26-2013 23:05:58.933 +0000 WARN TcpOutputProc - Cooked connection to ip=54.243.3.115:9997 timed out
11-26-2013 23:06:18.936 +0000 WARN TcpOutputProc - Cooked connection to ip=50.17.56.245:9997 timed out
11-26-2013 23:06:20.068 +0000 INFO TcpOutputProc - Connected to idx=54.224.46.188:9997 using ACK.
11-26-2013 23:06:58.937 +0000 WARN TcpOutputProc - Cooked connection to ip=23.22.208.232:9997 timed out
11-26-2013 23:07:18.940 +0000 WARN TcpOutputProc - Cooked connection to ip=107.20.29.58:9997 timed out
11-26-2013 23:07:20.942 +0000 INFO TcpOutputProc - Connected to idx=23.20.94.208:9997 using ACK.
11-26-2013 23:07:28.600 +0000 WARN TcpOutputProc - Shutdown timed out for 23.23.14.246:9997
11-26-2013 23:07:40.590 +0000 INFO TcpOutputProc - Connected to idx=54.224.135.13:9997 using ACK.

If I'm using splunk>storm then I won't be able to see indexers splunkd.log and metrics.log to see what is going on with the indexer, right? If so, where do I go from here?

My setup is as follows:
webserver nodes->splunkforwarder->splunkstorm

Both webserver nodes and splunkforwarder shows active forwards.

Any help is greatly appreciated.
Thank you.

Reply

phoffman_splunk
Splunk Employee
Splunk Employee
‎11-27-2013 05:35 AM

You can search the _internal index to see your splunkd and metrics logs. If you want anyone in particular,... index=_internal source=*metrics.log

0 Karma
Reply

phoffman_splunk
Splunk Employee
Splunk Employee
‎07-03-2013 06:45 AM

You should check in your indexers splunkd.log and metrics.log to see what is going on with your indexer. Can't tell much from looking at one side of the issue.

If you still don't get enough info you can crank up the logging on both sides to get more detail.

method of increasing logging on inputs and outputs: http://blogs.splunk.com/2008/09/22/enabling-debug-messages/

Reply
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...