Why do I get a massive amount of TcpOutputProc on my search head ?

[yoho]

I have a search head cluster (3 search heads) and an indexer cluster (3 indexers). More than 10% of splunkd.log (on my search heads) are produced by "TcpOutputProc". Is it an unusual amount ?

$ grep TcpOutputProc splunk/splunkd.log | wc -l
6780
$ wc -l splunk/splunkd.log
59586 splunk/splunkd.log
$ grep TcpOutputProc splunk/splunkd.log | head -5
02-04-2020 10:36:49.831 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.142:9997
02-04-2020 10:36:49.832 +0100 INFO  TcpOutputProc - Connected to idx=xxx.143:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.569 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.143:9997
02-04-2020 10:37:08.570 +0100 INFO  TcpOutputProc - Connected to idx=xxx.141:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.574 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.141:9997

[yoho]

Ok, I think logging each successful connection with 2 lines is a bit useless but well... thank you

[nickhills · Accepted solution]

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.