Deployment Architecture

Why do I get a massive amount of TcpOutputProc on my search head ?

yoho
Contributor

I have a search head cluster (3 search heads) and an indexer cluster (3 indexers). More than 10% of splunkd.log (on my search heads) are produced by "TcpOutputProc". Is it an unusual amount ?

$ grep TcpOutputProc splunk/splunkd.log | wc -l
6780
$ wc -l splunk/splunkd.log
59586 splunk/splunkd.log
$ grep TcpOutputProc splunk/splunkd.log | head -5
02-04-2020 10:36:49.831 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.142:9997
02-04-2020 10:36:49.832 +0100 INFO  TcpOutputProc - Connected to idx=xxx.143:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.569 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.143:9997
02-04-2020 10:37:08.570 +0100 INFO  TcpOutputProc - Connected to idx=xxx.141:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.574 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.141:9997
1 Solution

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

yoho
Contributor

Ok, I think logging each successful connection with 2 lines is a bit useless but well... thank you

0 Karma

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!