This is a diagram of Splunk components and network ports that are commonly used in a Splunk Enterprise environment. Firewall rules often need to be updated to allow communication on ports 8000, 8089, 9997, 514 and others.
Source files available here: http://downloads.jordan2000.com/splunk/
This is great.
For what it's worth, the cluster master / indexers communication is bidirectional (maybe double-headed arrow?), and port 9997 is primarily a tradition though I think the GUI provides that default. There's a support generated diagram somewhere that labells SSL-by-default differently from others, but that's synonymous with the default-port 8089 traffic.
Technically external apps can use REST API to any nodes in the system, but that's going to be only really useful for custom administration goals or custom troubleshooting goals typically. The search head is definitely the far most common target.
Thanks. The other direction of communication for the Cluster Master would be due to the Cluster Master polling members of the cluster, correct? (That would be in addition to the members themselves checking in with the Cluster Master)
Great diagram. Is there an updated one to include Search Head Clustering? New ports required are 8191 for the KV store, and a replication port chosen at implementation time (I have seen 8989 used) for search head cluster members to replicate data.
I note that this and the Splunk web GUI suggests port 8080 for indexer port replication.
However the CLI and conf documentation suggests to use port 9887.
Granted both will work however which is the Splunk sanctioned port 8080 or 9887.
There's really no sanctioned port, as such. 9887 is just an example of a port that you can use for the purpose.
As the documentation states, "You can specify any available, unused port as the replication port. Do not re-use the management or receiving ports."
@rob_jordan: Is the source of this image available anywhere? There are a couple discrepancies that should be fixes (as mentioned in the comments), but its the best diagram I have seen.