Why is our license master unreachable and Index replication not working?

[ramprakash]

Hi Splunkers,

One of my indexers is not able to communicate with the master properly. Previously, it was working fine. Due to this, index replication is not working properly.

Error on Indexer

Gui

In handler 'clusterslaveinfo': Slave is unable to handle request at this time. This means either slave is unable to communicate w/ master OR master does not have  peers added to the cluster. Check master dashboard and/or master_uri/secret settings.

Config File

Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers master=https: 
rv=0 actual_response_code=500 expected_response_code=201 status_line=HTTP/1.1 500 Internal Server Error [ event=addPeer status=retrying 
AddPeerRequest: { _id= active_bundle_id=B735 add_type=Initial-Add base_generation_id=0 forwarderdata_rcv_port=9100
 forwarderdata_use_ssl=0 mgmt_port=9089 name=7 register_forwarder_address= register_replication_address= 
register_search_address= replication_port=9200 replication_use_ssl=0 replications= server_name= } ].

02-13-2019 14:17:36.768 +0100 INFO  LicenseMgr - Initing LicenseMgr
02-13-2019 14:17:36.768 +0100 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
02-13-2019 14:17:36.768 +0100 INFO  LicenseMgr - StackMgr init complete...
02-13-2019 14:17:36.834 +0100 WARN  LMDirective - directive cmd=D_set_feature_state args='AWSMarketplace,DISABLED_DUE_TO_LICENSE' failed: reason='feature='AWSMarketplace' is invalid'
02-13-2019 14:17:36.834 +0100 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
02-13-2019 14:17:36.834 +0100 WARN  LMDirective - directive cmd=D_set_feature_state args='HideQuotaWarnings,DISABLED_DUE_TO_LICENSE' failed: reason='feature='HideQuotaWarnings' is invalid'
02-13-2019 14:17:36.834 +0100 INFO  LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
02-13-2019 14:17:36.835 +0100 WARN  LMDirective - directive cmd=D_set_feature_state args='SubgroupId,DISABLED_DUE_TO_LICENSE' failed: reason='feature='SubgroupId' is invalid'
02-13-2019 14:17:36.836 +0100 INFO  LicenseMgr - Tracker init complete...
02-13-2019 14:17:38.768 +0100 INFO  WatchedFile - Will begin reading at offset=16950 for file='/APPLICATIONS/splunk/var/log/splunk/license_usage.log'.

[woodcock]

Definitely open a support case on this.

[nickhills]

Can you repost the log message in a codeblock (the icon that looks like 101010), as it looks like the HTML formatter has removed some of the data.

Also, can you include the logs which specify the LMConfig parameters too