Splunk Search

TcpOutputProc - Cooked connection to Forwarder IP:9997 timed out

smdasim
Explorer

Hi,

We have a indexer{2 indexers] in our environment, 2 fowarder and 1 search heads.
I am seeing below output on Search head .

TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out

TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out

Please advice how can i debug at each level to figure out the issue .
Data is not reaching to newly created web index.

Regrads
smdasim

Tags (1)
0 Karma

nickhills
Ultra Champion

Just to be clear:

"TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out"

Is not necessarily a problem - It can just indicate a network connection between splunk components has stopped sending data and has naturally timed out - as would any other TCP connection.

You could well be chasing a red-herring - if your forwarders are not able to communicate with your indexers, you wouldn't see any internal logs.
If you are seeing internal logs, its almost certainly config related, and not a network issue.

I would check that your indexes are properly defined, and that your inputs specify the correct index definitions

If my comment helps, please give it a thumbs up!
0 Karma

smdasim
Explorer

Solution :Create indexes and give user roles on search head and indexers as shown below

https://developers.perfectomobile.com/display/TT/Splunk+-+Creating+your+Index

bpadmanbhachari
Splunk Employee
Splunk Employee

Cooked connection denotes communications between two Splunk nodes as opposed to Raw connections which refer to non-Splunk nodes passing their data to Splunk.

First we need to ensure that the forwarder is listening on the correct port. To determine this we simply issue a netstat:

netstat –an | grep 9997

If the forwarder is listening properly you should see a result similar to:

Tcp 0 0 0.0.0.0:9997 0.0.0.0:* LIST

This tells us that the forwarder is in listen mode for port 9998.

Now that we’ve determined we’re listening on the correct port we need to test the communications path between the forwarder and indexer. To do this we attempt to open a telnet session to the indexer from the forwarder:

telnet indexername.domainname.com 9997

If the port is available the connection should be successful almost immediately. Should the connection fail we’ll want to try another port to determine if we have a port availability issue or something more. So next we’ll attempt to telnet to port 8089 (Splunk management port which should always be open on an indexer):

telnet indexername.domainname.com 8089

If the connection is successful you should see something close to the following:

Connected to indexername.domainname.com

Once we’ve made the successful connection, we now know that there is an open port issue with port 9998 to the indexer. Now the question remains, is this an issue at the firewall layer or within a local firewall such as iptables on the indexer itself. To determine this, we SSH into the indexer and attempt to telnet back onto ourselves via port 9998.

telnet localhost 9997

If the problem lies with a local firewall such as iptables, you will receive an error similar to the following:

telnet: connect to address ::1: Connection refused

So what does this tell us? The refusal locally tells us that port 9998 has not been opened via the local firewall and this is the source of our issue. To resolve we simply open port 9998 via iptables (or your local firewall) to resolve.

iptables –D INPUT –p tcp –dport 9997 –j DROP

service iptables save

If the telnet connection to localhost connects successfully, you have determined that the communications issue is tied to the fact that port 9998 needs to be opened at the network firewall layer to allow communications with the forwarder.

JuGuSm
Path Finder

"This tells us that the forwarder is in listen mode for port 9997." (not 9998)

0 Karma

smdasim
Explorer

netstats -a | grep 9997 does not return any output on fowarder or indexer . The below results "place-6" is shown .please advice

[root@ip-172-31-17-211 ~]# cat /etc/services | grep 9997
palace-6 9997/tcp # Palace-6
palace-6 9997/udp # Palace-6
[root@ip-172-31-17-211 ~]#

[root@ip-172-31-17-211 ~]# netstat -a | grep tcp
tcp 0 0 :ssh *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:smtp : LISTEN

tcp 0 0 ip-172-31-17-211.ap-s:64804 ip-172-31-29-204.a:palace-6 ESTABLISHED
tcp 0 0 ip-172-31-17-211.ap-s:46794 ip-172-31-29-189.a:palace-6 TIME_WAIT

tcp 0 0 ip-172-31-17-211.ap-s:14322 ip-172-31-19-169.ap-so:8089 ESTABLISHED
tcp 0 40 ip-172-31-17-211.ap-sou:ssh 27-32-105-2.static.tp:12830 ESTABLISHED

0 Karma

smdasim
Explorer

is the Port blocked ? netstat –an | grep 9997 command returns no output at forwarder or indexer level
please advice

netstats output at forwarder
root@ip-172-31-26-242 bin]# netstat -a |grep 9997
[root@ip-172-31-26-242 bin]#
netstats output at indexer

[root@ip-172-31-29-189 ~]# netstat -a | grep 9997
[root@ip-172-31-29-189 ~]#

Forwarder config

[root@ip-172-31-26-242 bin]# ./splunk list forward-server
Your session is invalid. Please login.
Splunk username: admin
Password:
Active forwards:
172.31.29.204:9997
Configured but inactive forwards:
172.31.29.189:9997
[root@ip-172-31-26-242 bin]#

[root@ip-172-31-26-242 bin]# ./splunk show deploy-poll
Deployment Server URI is set to "172.31.19.169:8089".
[root@ip-172-31-26-242 bin]#

Fowarder output

[root@ip-172-31-17-211 local]# telnet 172.31.29.189 9997
Trying 172.31.29.189...
Connected to 172.31.29.189.
Escape character is '^]'.

[root@ip-172-31-17-211 local]# telnet 172.31.29.189 8089
Trying 172.31.29.189...
Connected to 172.31.29.189.
Escape character is '^]'.

Indexer output

[root@ip-172-31-29-189 ~]# telnet localhost 9997
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

0 Karma

smdasim
Explorer

hi,

please list of connection with the command list inputstatus ( tcp_cooked:listenerports 9997)

[root@ip-172-31-29-189 bin]# ./splunk list inputstatus
Your session is invalid. Please login.
Splunk username: admin
Password:
Cooked:tcp :
9997:172.31.17.211:8089
time opened = 2018-08-16T21:57:14+0000

9997:172.31.26.242:8089
    time opened = 2018-08-16T21:55:31+0000

tcp

ExecProcessor:exec commands :
$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/instrumentation.py

$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/on_splunk_start.py
    exit status description = exited with code 0
    time closed = 2018-08-16T21:55:17+0000
    time opened = 2018-08-16T21:55:17+0000

$SPLUNK_HOME/etc/apps/splunk_instrumentation/bin/schedule_delete.py

./bin/collector.path
    time opened = 2018-08-16T21:55:14+0000

./bin/dmc_config.py
    exit status description = exited with code 0
    time closed = 2018-08-16T21:55:20+0000
    time opened = 2018-08-16T21:55:19+0000

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/introspection
    type = directory

$SPLUNK_HOME/var/log/splunk
    type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    type = directory

$SPLUNK_HOME/var/spool/splunk/...stash_new
    type = directory

/opt/splunk/var/log/introspection/disk_objects.log
    file position = 158098
    file size = 158098
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/introspection/http_event_collector_metrics.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100
    type = finished reading

/opt/splunk/var/log/introspection/kvstore.log
    file position = 14684499
    file size = 14684499
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/introspection/resource_usage.log
    file position = 12051661
    file size = 12051661
    parent = $SPLUNK_HOME/var/log/introspection
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/audit.log
    file position = 1941241
    file size = 1941241
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/btool.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/conf.log
    file position = 1184
    file size = 1184
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/django_access.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/django_error.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/django_service.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/export_metrics.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/first_install.log
    file position = 70
    file size = 70
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/health.log
    file position = 713440
    file size = 713440
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/license_usage.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/license_usage_summary.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/metrics.log
    file position = 15724751
    file size = 15724751
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/mongod.log
    file position = 62397
    file size = 62397
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/python.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/remote_searches.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/scheduler.log
    file position = 9482
    file size = 9482
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/searchhistory.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/splunk_archiver.log
    file position = 1088
    file size = 1088
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunk_instrumentation.log
    file position = 0
    file size = 0
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100
    type = finished reading

/opt/splunk/var/log/splunk/splunkd-utility.log
    file position = 4953
    file size = 4953
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd.log
    file position = 424860
    file size = 424860
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_access.log
    file position = 237905
    file size = 237905
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/splunkd_stderr.log
    file position = 454
    file size = 454
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_stdout.log
    file position = 2042
    file size = 2042
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = finished reading

/opt/splunk/var/log/splunk/splunkd_ui_access.log
    file position = 165442
    file size = 165442
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/web_access.log
    file position = 46960
    file size = 46960
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

/opt/splunk/var/log/splunk/web_service.log
    file position = 85686
    file size = 85686
    parent = $SPLUNK_HOME/var/log/splunk
    percent = 100.00
    type = open file

tcp_cooked:listenerports :
9997

[root@ip-172-31-29-189 bin]#

0 Karma

smdasim
Explorer

Indexer 1

[root@ip-172-31-29-189 ~]# netstat --tcp --listen --numeric-ports
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 :8191 *: LISTEN

tcp 0 0 :8000 *: LISTEN

tcp 0 0 localhost:8065 : LISTEN

tcp 0 0 :9997 *: LISTEN

tcp 0 0 :22 *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:25 : LISTEN

[root@ip-172-31-29-189 ~]#
[root@ip-172-31-29-189 ~]#

Index2

[root@ip-172-31-29-204 ~]# netstat --tcp --listen --numeric-ports
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 :8191 *: LISTEN

tcp 0 0 :8000 *: LISTEN

tcp 0 0 localhost:8065 : LISTEN

tcp 0 0 :9997 *: LISTEN

tcp 0 0 :22 *: LISTEN

tcp 0 0 :8089 *: LISTEN

tcp 0 0 localhost:25 : LISTEN

[root@ip-172-31-29-204 ~]#

Fwd1

[root@ip-172-31-17-211 ~]# /opt/splunkforwarder/bin/splunk show deploy-poll
Deployment Server URI is set to "[172.31.29:189]:8089".
[root@ip-172-31-17-211 ~]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
172.31.29.204:9997
Configured but inactive forwards:
172.31.29.189:9997
[root@ip-172-31-17-211 ~]#

Fwd2

[root@ip-172-31-26-242 ~]# /opt/splunkforwarder/bin/splunk show deploy-poll
Deployment Server URI is set to "172.31.29.189:8089".
[root@ip-172-31-26-242 ~]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
172.31.29.189:9997
Configured but inactive forwards:
172.31.29.204:9997
[root@ip-172-31-26-242 ~]#

Ports are open and listening (FWD-to indexer)

[root@ip-172-31-17-211 ~]# nc -z 172.31.29.189 9997; echo $?
Connection to 172.31.29.189 9997 port [tcp/palace-6] succeeded!
0
[root@ip-172-31-17-211 ~]# nc -z 172.31.29.204 9997; echo $?
Connection to 172.31.29.204 9997 port [tcp/palace-6] succeeded!
0

0 Karma

smdasim
Explorer

Brain,
There are forwarder IP address and my data is not reaching newly created web index on the search head and when I look on the searched head it show

TcpOutputProc - Cooked connection to FWD ip=x.x.x.x:9997 timed out

TcpOutputProc - Cooked connection to FWD ip=x.x.x.x:9997 timed out

Regards
smdasim

0 Karma

brian_rampley
Path Finder

Which servers in your environment does the message match? As in, do the IPs match your search heads, indexers, forwarders, or none of the above.

0 Karma

smdasim
Explorer

they match the forwarder IP's

0 Karma
Get Updates on the Splunk Community!

Discover Powerful New Features in Splunk Cloud Platform: Enhanced Analytics, ...

Hey Splunky people! We are excited to share the latest updates in Splunk Cloud Platform 9.3.2408. In this ...

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...