Deployment Architecture

Is it normal for an indexer cluster master to connect to peers on odd ports?

tkw03
Communicator

I was troubleshooting why peers show as "Pending" often in the cluster master web UI. In troubleshooting I ran 'ss |less' and via TCP, I found the master connecting on odd ports and vice versa. Here's a "sanitized" example:

Netid  State      Recv-Q Send-Q Local Address:Port                 Peer Address:Port 
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.06:41346
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.04:47714
tcp    ESTAB      0      0      172.indexercluster.master.ip:40738                172.indexercluster.member.015:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:53218                172.indexercluster.member.010:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:57761                172.indexercluster.member.018:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:60002                172.indexercluster.member.012:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:54722                172.indexercluster.member.021:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:57434                172.indexercluster.member.014:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.010:40392
tcp    ESTAB      0      0      172.indexercluster.master.ip:57484                172.indexercluster.member.014:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.018:39212
tcp    ESTAB      0      0      172.indexercluster.master.ip:44492                172.indexercluster.member.013:8089

Is this normal communication or something strange?

Not sure I've noticed this before, so I wanted to see if anyone else has seen this.

Thanks

0 Karma
1 Solution

nickhills
Ultra Champion

With a TCP connection the 'client' connects to the 'server' on a known or target port.
In the case of Splunk this is 8089. However since TCP is a bidirectional protocol it needs to tell the server which port to reply back on.
In most c/s architectures, the port the client chooses is a random 'high-number' port - 41346 from the top line is one such example.

The top line, is a connection from 'member6' using 41346 as its src port to the dest port 8089 on the master.

tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.06:41346

The third line, is a connection from the master with src port 40738 to the dest port 8089 on 'member15'

tcp    ESTAB      0      0      172.indexercluster.master.ip:40738                172.indexercluster.member.015:8089

What your seeing is totally normal TCP communication patterns.

If my comment helps, please give it a thumbs up!

View solution in original post

nickhills
Ultra Champion

With a TCP connection the 'client' connects to the 'server' on a known or target port.
In the case of Splunk this is 8089. However since TCP is a bidirectional protocol it needs to tell the server which port to reply back on.
In most c/s architectures, the port the client chooses is a random 'high-number' port - 41346 from the top line is one such example.

The top line, is a connection from 'member6' using 41346 as its src port to the dest port 8089 on the master.

tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.06:41346

The third line, is a connection from the master with src port 40738 to the dest port 8089 on 'member15'

tcp    ESTAB      0      0      172.indexercluster.master.ip:40738                172.indexercluster.member.015:8089

What your seeing is totally normal TCP communication patterns.

If my comment helps, please give it a thumbs up!

tkw03
Communicator

Thanks, was just making sure it wasn't something abnormal

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...