Splunk Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

TcpOutputProc - Cooked connection to Forwarder IP:9997 timed out

smdasim
Explorer
‎08-12-2018 04:41 PM

Hi,

We have a indexer{2 indexers] in our environment, 2 fowarder and 1 search heads.
I am seeing below output on Search head .

TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out

TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out

Please advice how can i debug at each level to figure out the issue .
Data is not reaching to newly created web index.

Regrads
smdasim

Tags (1)
0 Karma
Reply

nickhills
Ultra Champion
‎01-07-2019 08:24 AM

Just to be clear:

"TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out"

Is not necessarily a problem - It can just indicate a network connection between splunk components has stopped sending data and has naturally timed out - as would any other TCP connection.

You could well be chasing a red-herring - if your forwarders are not able to communicate with your indexers, you wouldn't see any internal logs.
If you are seeing internal logs, its almost certainly config related, and not a network issue.

I would check that your indexes are properly defined, and that your inputs specify the correct index definitions

If my comment helps, please give it a thumbs up!
0 Karma
Reply

smdasim
Explorer
‎08-17-2018 07:31 PM

Solution :Create indexes and give user roles on search head and indexers as shown below

https://developers.perfectomobile.com/display/TT/Splunk+-+Creating+your+Index

Reply

bpadmanbhachari
Splunk Employee
Splunk Employee
‎08-16-2018 12:36 AM

Cooked connection denotes communications between two Splunk nodes as opposed to Raw connections which refer to non-Splunk nodes passing their data to Splunk.

First we need to ensure that the forwarder is listening on the correct port. To determine this we simply issue a netstat:

netstat –an | grep 9997

If the forwarder is listening properly you should see a result similar to:

Tcp 0 0 0.0.0.0:9997 0.0.0.0:* LIST

This tells us that the forwarder is in listen mode for port 9998.

Now that we’ve determined we’re listening on the correct port we need to test the communications path between the forwarder and indexer. To do this we attempt to open a telnet session to the indexer from the forwarder:

telnet indexername.domainname.com 9997

If the port is available the connection should be successful almost immediately. Should the connection fail we’ll want to try another port to determine if we have a port availability issue or something more. So next we’ll attempt to telnet to port 8089 (Splunk management port which should always be open on an indexer):

telnet indexername.domainname.com 8089

If the connection is successful you should see something close to the following:

Connected to indexername.domainname.com

Once we’ve made the successful connection, we now know that there is an open port issue with port 9998 to the indexer. Now the question remains, is this an issue at the firewall layer or within a local firewall such as iptables on the indexer itself. To determine this, we SSH into the indexer and attempt to telnet back onto ourselves via port 9998.

telnet localhost 9997

If the problem lies with a local firewall such as iptables, you will receive an error similar to the following:

telnet: connect to address ::1: Connection refused

So what does this tell us? The refusal locally tells us that port 9998 has not been opened via the local firewall and this is the source of our issue. To resolve we simply open port 9998 via iptables (or your local firewall) to resolve.

iptables –D INPUT –p tcp –dport 9997 –j DROP

service iptables save

If the telnet connection to localhost connects successfully, you have determined that the communications issue is tied to the fact that port 9998 needs to be opened at the network firewall layer to allow communications with the forwarder.

Reply

JuGuSm
Path Finder
‎01-07-2019 08:19 AM

"This tells us that the forwarder is in listen mode for port 9997." (not 9998)

0 Karma
Reply

smdasim
Explorer
‎08-17-2018 05:27 PM

netstats -a | grep 9997 does not return any output on fowarder or indexer . The below results "place-6" is shown .please advice

[root@ip-172-31-17-211 ~]# cat /etc/services | grep 9997
palace-6 9997/tcp # Palace-6
palace-6 9997/udp # Palace-6
[root@ip-172-31-17-211 ~]#

[root@ip-172-31-17-211 ~]# netstat -a | grep tcp
tcp 0 0 :ssh *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:smtp : LISTEN

tcp 0 0 ip-172-31-17-211.ap-s:64804 ip-172-31-29-204.a:palace-6 ESTABLISHED
tcp 0 0 ip-172-31-17-211.ap-s:46794 ip-172-31-29-189.a:palace-6 TIME_WAIT

tcp 0 0 ip-172-31-17-211.ap-s:14322 ip-172-31-19-169.ap-so:8089 ESTABLISHED
tcp 0 40 ip-172-31-17-211.ap-sou:ssh 27-32-105-2.static.tp:12830 ESTABLISHED

0 Karma
Reply

smdasim
Explorer
‎08-17-2018 01:53 PM

is the Port blocked ? netstat –an | grep 9997 command returns no output at forwarder or indexer level
please advice

netstats output at forwarder
root@ip-172-31-26-242 bin]# netstat -a |grep 9997
[root@ip-172-31-26-242 bin]#
netstats output at indexer

[root@ip-172-31-29-189 ~]# netstat -a | grep 9997
[root@ip-172-31-29-189 ~]#

Forwarder config

[root@ip-172-31-26-242 bin]# ./splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Password:
Active forwards:
172.31.29.204:9997
Configured but inactive forwards:
172.31.29.189:9997
[root@ip-172-31-26-242 bin]#

[root@ip-172-31-26-242 bin]# ./splunk show deploy-poll
Deployment Server URI is set to "172.31.19.169:8089".
[root@ip-172-31-26-242 bin]#

Fowarder output

[root@ip-172-31-17-211 local]# telnet 172.31.29.189 9997
Trying 172.31.29.189...
Connected to 172.31.29.189.
Escape character is '^]'.

[root@ip-172-31-17-211 local]# telnet 172.31.29.189 8089
Trying 172.31.29.189...
Connected to 172.31.29.189.
Escape character is '^]'.

Indexer output

[root@ip-172-31-29-189 ~]# telnet localhost 9997
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

0 Karma
Reply

smdasim
Explorer
‎08-16-2018 03:19 PM

hi,

please list of connection with the command list inputstatus ( tcp_cooked:listenerports 9997)

[root@ip-172-31-29-189 bin]# ./splunk list inputstatus
Your session is invalid. Please login.
Splunk username: admin
Password:
Cooked:tcp :
9997:172.31.17.211:8089
time opened = 2018-08-16T21:57:14+0000

9997:172.31.26.242:8089
    time opened = 2018-08-16T21:55:31+0000

tcp

ExecProcessor:exec commands :
$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/instrumentation.py

$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/on_splunk_start.py
    exit status description = exited with code 0
    time closed = 2018-08-16T21:55:17+0000
    time opened = 2018-08-16T21:55:17+0000

$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/schedule_delete.py

./bin/collector.path
    time opened = 2018-08-16T21:55:14+0000

./bin/dmc_config.py
    exit status description = exited with code 0
    time closed = 2018-08-16T21:55:20+0000
    time opened = 2018-08-16T21:55:19+0000

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/introspection
    type = directory

$SPLUNK_HOME/var/log/splunk
    type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    type = directory

$SPLUNK_HOME/var/spool/splunk/...stash_new
    type = directory

/opt/splunk/var/log/introspection/disk_objects.log
    file position = 158098
    file size = 158098
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/introspection/http_event_collector_metrics.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100
    type = finished reading

/opt/splunk/var/log/introspection/kvstore.log
    file position = 14684499
    file size = 14684499
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/introspection/resource_usage.log
    file position = 12051661
    file size = 12051661
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/audit.log
    file position = 1941241
    file size = 1941241
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/btool.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/conf.log
    file position = 1184
    file size = 1184
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/django_access.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/django_error.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/django_service.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/export_metrics.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/first_install.log
    file position = 70
    file size = 70
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/health.log
    file position = 713440
    file size = 713440
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/license_usage.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/license_usage_summary.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/metrics.log
    file position = 15724751
    file size = 15724751
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/mongod.log
    file position = 62397
    file size = 62397
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/python.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/remote_searches.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/scheduler.log
    file position = 9482
    file size = 9482
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/searchhistory.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/splunk_archiver.log
    file position = 1088
    file size = 1088
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunk_instrumentation.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/splunkd-utility.log
    file position = 4953
    file size = 4953
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd.log
    file position = 424860
    file size = 424860
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_access.log
    file position = 237905
    file size = 237905
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/splunkd_stderr.log
    file position = 454
    file size = 454
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_stdout.log
    file position = 2042
    file size = 2042
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_ui_access.log
    file position = 165442
    file size = 165442
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/web_access.log
    file position = 46960
    file size = 46960
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/web_service.log
    file position = 85686
    file size = 85686
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

tcp_cooked:listenerports :
9997

[root@ip-172-31-29-189 bin]#

0 Karma
Reply

smdasim
Explorer
‎08-16-2018 03:38 PM

Indexer 1

[root@ip-172-31-29-189 ~]# netstat --tcp --listen --numeric-ports
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 :8191 *: LISTEN

tcp 0 0 :8000 *: LISTEN

tcp 0 0 localhost:8065 : LISTEN

tcp 0 0 :9997 *: LISTEN

tcp 0 0 :22 *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:25 : LISTEN

[root@ip-172-31-29-189 ~]#
[root@ip-172-31-29-189 ~]#

Index2

[root@ip-172-31-29-204 ~]# netstat --tcp --listen --numeric-ports
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 :8191 *: LISTEN

tcp 0 0 :8000 *: LISTEN

tcp 0 0 localhost:8065 : LISTEN

tcp 0 0 :9997 *: LISTEN

tcp 0 0 :22 *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:25 : LISTEN

[root@ip-172-31-29-204 ~]#

Fwd1

[root@ip-172-31-17-211 ~]# /opt/splunkforwarder/bin/splunk show deploy-poll
Deployment Server URI is set to "[172.31.29:189]:8089".
[root@ip-172-31-17-211 ~]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
172.31.29.204:9997
Configured but inactive forwards:
172.31.29.189:9997
[root@ip-172-31-17-211 ~]#

Fwd2

[root@ip-172-31-26-242 ~]# /opt/splunkforwarder/bin/splunk show deploy-poll
Deployment Server URI is set to "172.31.29.189:8089".
[root@ip-172-31-26-242 ~]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
172.31.29.189:9997
Configured but inactive forwards:
172.31.29.204:9997
[root@ip-172-31-26-242 ~]#

Ports are open and listening (FWD-to indexer)

[root@ip-172-31-17-211 ~]# nc -z 172.31.29.189 9997; echo $?
Connection to 172.31.29.189 9997 port [tcp/palace-6] succeeded!
0
[root@ip-172-31-17-211 ~]# nc -z 172.31.29.204 9997; echo $?
Connection to 172.31.29.204 9997 port [tcp/palace-6] succeeded!
0

0 Karma
Reply

smdasim
Explorer
‎08-16-2018 12:36 AM

Brain,
There are forwarder IP address and my data is not reaching newly created web index on the search head and when I look on the searched head it show

TcpOutputProc - Cooked connection to FWD ip=x.x.x.x:9997 timed out

TcpOutputProc - Cooked connection to FWD ip=x.x.x.x:9997 timed out

Regards
smdasim

0 Karma
Reply

brian_rampley
Path Finder
‎08-13-2018 12:01 AM

Which servers in your environment does the message match? As in, do the IPs match your search heads, indexers, forwarders, or none of the above.

0 Karma
Reply

smdasim
Explorer
‎08-15-2018 11:30 PM

they match the forwarder IP's

0 Karma
Reply
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...