Deployment Architecture

Why do I get a massive amount of TcpOutputProc on my search head ?

yoho
Contributor

I have a search head cluster (3 search heads) and an indexer cluster (3 indexers). More than 10% of splunkd.log (on my search heads) are produced by "TcpOutputProc". Is it an unusual amount ?

$ grep TcpOutputProc splunk/splunkd.log | wc -l
6780
$ wc -l splunk/splunkd.log
59586 splunk/splunkd.log
$ grep TcpOutputProc splunk/splunkd.log | head -5
02-04-2020 10:36:49.831 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.142:9997
02-04-2020 10:36:49.832 +0100 INFO  TcpOutputProc - Connected to idx=xxx.143:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.569 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.143:9997
02-04-2020 10:37:08.570 +0100 INFO  TcpOutputProc - Connected to idx=xxx.141:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.574 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.141:9997
1 Solution

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

yoho
Contributor

Ok, I think logging each successful connection with 2 lines is a bit useless but well... thank you

0 Karma

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...