topic Re: Splunk closing TCP port 9997 (forwarder port) in Getting Data In

Splunk closing TCP port 9997 (forwarder port)

caphrim007 — Thu, 12 Jul 2012 02:14:40 GMT

I upgraded to 4.3.3 on an indexer that never had any problems before this point in time and now the indexer is dropping all forwarded events on the floor with messages like this

07-11-2012 12:44:17.568 -0500 INFO TcpInputProc - Stopping IPv4 port 9997
07-11-2012 12:44:17.568 -0500 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds

I've seen similar questions appear like this on splunkanswers, but the suggested resolutions (involving fishbucket) dont seem to apply to my case?

I turned on splunk debugging, but it doesn't lead me to any better conclusions.

What queues is it referring to? The box is ripe with CPU, disk, and RAM. It cant possibly be overloaded; it's not doing anything.

Support is being a lame duck; taking their time staring at walls. In the meantime my primary splunk indexer is not indexing anything because it's not receiving anything from the forwarders.

Does anyone have any clues as to where I could look? If it's not resolved by tomorrow I'm re-installing splunk on the primary indexer as this is not something that can wait.

Thanks in advance for any help and guidance you can provide.

Re: Splunk closing TCP port 9997 (forwarder port)

hexx — Thu, 12 Jul 2012 04:59:13 GMT

The queues that are mentioned by that message are those that lead into the data pipelines where splunkd shapes your data into events before indexing those on disk.

This message would indicate that there is a bottleneck in one of those pipelines, which causes the queue that feeds it and all queues upstream to fill up, all the way to the queue that accepts incoming events from forwarders (splunktcpin).

This is obviously undesirable, but keep in mind that your forwarder events are not being dropped. Instead, the forwarders will pause their data inputs and resume once the indexer is able to process data again.

When seeing such a message, the first thing that you should do is to determine the fill percentage of the queues leading to the 4 main data pipelines : parsing -> merging -> typing -> indexing.

By determining which is the most downstream queue to be saturated, you can get an idea of why there is a bottleneck there.

A simple way to gain visibility of the state of event-processing queues is to use the "indexing performance" view of the Splunk on Splunk app. For details on how to install the app, check this Splunk Answer.

If you can post a screenshot showing the panels of that view, I can try to help you further.

Incidentally, what is the case number that you opened with Splunk support? I can check in on it for you.

Re: Splunk closing TCP port 9997 (forwarder port)

caphrim007 — Thu, 12 Jul 2012 11:30:24 GMT

Appears I have it installed, but when going to use it, I get this error.

Splunk encountered the following unknown module: "sosFTR" . The view may not load properly

Re: Splunk closing TCP port 9997 (forwarder port)

caphrim007 — Thu, 12 Jul 2012 11:37:22 GMT

Sideview Utils. Next time I'll read before asking

Re: Splunk closing TCP port 9997 (forwarder port)

mikelanghorst — Thu, 12 Jul 2012 17:40:17 GMT

It's a shot in the dark without more information, but I had this issue before. Are you using the deployment server in your environment? Is it possible your forwarders' outputs.conf got deployed to your indexer?

On the indexer:
./splunk cmd btool outputs list --debug

See if you're somehow looping your inputs back to itself.

Re: Splunk closing TCP port 9997 (forwarder port)

hexx — Thu, 12 Jul 2012 18:03:58 GMT

That would be consistent with the high-level symptom described.

Re: Splunk closing TCP port 9997 (forwarder port)

KpiBuff — Thu, 05 Feb 2015 22:17:20 GMT

I try to avoid staring at walls whenever I can 😉

Re: Splunk closing TCP port 9997 (forwarder port)

k_harini — Thu, 25 May 2017 14:11:59 GMT

What was the issue? Please help.. We are facing same issue.. If this is resolved, can you please give snippet of inputs. Conf and output. Conf files..

Re: Splunk closing TCP port 9997 (forwarder port)

k_harini — Sat, 27 May 2017 02:11:49 GMT

Tat was the issue for me.. Looping back to itself