We have SH cluster of 3 SH, where enterprise security notable are not same on all 3 SH enterprise security. And further when we check for last 15 min internal data that also vary with significant number (5 K to 10 k) than other 2 SH Member.
@AShwin1119- I think its the same question here I have answered - https://community.splunk.com/t5/Monitoring-Splunk/indexer-cluster-to-SH-cluster-replication-issue/m-...
I think you are not forwarding the SH data to Indexers.
* Which is compulsory when you are using SHC.
* And best-practice in all SHs.
https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/Forwardsearchheaddata
I hope this helps!!! Kindly upvote if it does!!!!
OK. Check shcluster-status. Check splunkd.log on those instances (and mongodb.log). If the state of the SHC is not in sync... that means something is off with replication or the overall cluster health.
Hi @AShwin1119 ,
it's really strange, probably therewas some replication issue, did you checked the status of replication in the Cluster Manager Console?
Had you some stop of one or more of the indexers?
Have you a multisite or a single site Indexers Cluster?
If it's all OK, open a case to Splunk Support.
Ciao.
Giuseppe