How come replication isn't working on the Index cluster after a reboot?

[christopherr_sp]

We had to shut down one of the machines and create a new one. The cluster replication between the new and old ones does not work after a reboot.

The error message that was found in the splunkd.log files was:

ERROR RetryableClientTransaction - transactionDone(): transactionId=0x7fda3f101000
rTxnId=0x7fda3c5fe4d0 success=N HTTP-statusCode=404 HTTP-statusDescription=Not Found retry=N
no_retry_reason="transaction had fatal error" 

[christopherr_sp · Accepted solution]

It was found that Splunk OnPremise was installed on AWS and all of the ports were closed.

Once port 8089 was opened replication resumed.

The following Documentation Enhancement Request has been raised to be considered in a future release of Splunk.

SPL-163427 Enhancement Request to check all ports are open in an Indexer Cluster/Search Head Cluster environment

The default ports that Splunk uses are below.

What are the ports that I need to open?

https://answers.splunk.com/answers/58888/what-are-the-ports-that-i-need-to-open.html