Trying to add an indexer on AWS, how to resolve error "Master is down! Make sure pass4SymmKey is matching if master is running."?

[wangsimingxaxis]

I created a Splunk environment on AWS by using Splunk AMI.

1 master
2 search heads
3 indexers

They are in the same subnet and all ports are opened.

When I configure an indexer, I get below errors which tell me this indexer cannot connect to master

06-27-2016 14:29:49.849 +0000 WARN  CMMasterProxy - Master is down! Make sure pass4SymmKey is matching if master is running.
06-27-2016 14:29:49.849 +0000 WARN  CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=xxx.xx.xx.xxx:8089 rv=0 actual_response_code=500 expected_response_code=201 status_line="Internal Server Error" socket_error="No error" [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=C6371A0987CB1E656564B016E2208682 add_type=Initial-Add base_generation_id=0 latest_bundle_id=C6371A0987CB1E656564B016E2208682 mgmt_port=8089 name=36DF02F3-F116-4DFC-B667-A005D906C244 register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name=ip-172-31-10-189 site=default splunk_version=6.4.0 splunkd_build_number=f2c836328108 status=Up } ].
06-27-2016 14:39:41.631 +0000 WARN  AuditTrailManager - skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied.  Check disk space and other issues that may cause indexer to block

Any advice?

[ryanoconnor]

The first error is probably the first thing to check. Do you if pass4SymmKey is the same on all of your cluster members?

[wangsimingxaxis]

And I think if the security key does not match, it will not allow me to go to next step to reboot splunk console.

Because when I try to put a different security key , it rejected me to go to next and give error on the screen .

[wangsimingxaxis]

yes, it is I already use the same key to add Search head already without any issues