indexer cluster to SH cluster replication issue.

[AShwin1119]

we have a SH cluster with 3 SH which is collecting data with indexer cluster having 3 indexers. Now the problem is data present in the each indexer is not properly replicating in all 3 SH, example if we check for last 15 min _internal data on each SH then number of event is different by 1k to 5 k.

And if I create dashboard in SH then this is getting replicated properly in between the SH.

because of this issue in enterprise security notable is showing different in each SH.

[VatsalJagani]

@AShwin1119- I think you are not forwarding the SH data to Indexers.

* Which is compulsory when you are using SHC.

* And best-practice in all SHs.

https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/Forwardsearchheaddata

I hope this helps!! Please upvote if it helps!!!

[isoutamo]

Hi

when you have a distributed environment it’s mandatory that all servers send their internal logs into indexer cluster as @VatsalJagani already pointed! Without that it’s really hard to debug and monitoring this environment. You should also have MC and LM setting up in some node. Best to have own instance for those or use e.g. SH deployer for those. There are list of combinations which are allowed and supported by splunk in doc.splunk.com.

r. Ismo

[kiran_panchavat]

@AShwin1119

First, it's important to understand that the data needs to replicate properly across the indexers. When you search for data from the search head, it doesn't directly query the indexers. Instead, the search head first contacts the cluster master, which checks which indexers are available and retrieves the results from them.

If the replication and search factors are correctly configured on the cluster master, your environment should be functioning properly.

The data may be indexed on one indexer but not fully replicated across all indexers in the cluster or between the SHs. If the indexers are not properly replicating data to all search heads in a timely manner, you may see discrepancies in event counts when searching.

Please monitor your environment using the Monitoring Console, including the search heads, indexers, and other components. How can you ensure that the same notable events are visible across all search heads? If possible, could you provide a screenshot?