Deployment Architecture

Why do I get a massive amount of TcpOutputProc on my search head ?

yoho
Contributor

I have a search head cluster (3 search heads) and an indexer cluster (3 indexers). More than 10% of splunkd.log (on my search heads) are produced by "TcpOutputProc". Is it an unusual amount ?

$ grep TcpOutputProc splunk/splunkd.log | wc -l
6780
$ wc -l splunk/splunkd.log
59586 splunk/splunkd.log
$ grep TcpOutputProc splunk/splunkd.log | head -5
02-04-2020 10:36:49.831 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.142:9997
02-04-2020 10:36:49.832 +0100 INFO  TcpOutputProc - Connected to idx=xxx.143:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.569 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.143:9997
02-04-2020 10:37:08.570 +0100 INFO  TcpOutputProc - Connected to idx=xxx.141:9997, pset=0, reuse=0. using ACK.
02-04-2020 10:37:08.574 +0100 INFO  TcpOutputProc - Closing stream for idx=xxx.141:9997
1 Solution

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

yoho
Contributor

Ok, I think logging each successful connection with 2 lines is a bit useless but well... thank you

0 Karma

nickhills
Ultra Champion

That is the log of your SH sending its events to the indexers.

The more users/searches you have the more audit/internal events you generate, so the more events need to be sent to the indexers.
(Also there are logs relating to the SHC which need to be indexed, so even a quiet shc will have lots of these)

Each of those connections is an event stream from the SH beginning with "connected", and ending with "closed"
You will see that they normally cycle around all your indexers, so expect to see your indexer IP's in constant rotation.
Although this looks like a lot, each connection is opened and then gracefully closed, so its not using a huge number of connections at any one time.

Totally Normal.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...