Monitoring Splunk

[Multiple Clusters] search not returning from new cluster

sylim_splunk
Splunk Employee
Splunk Employee

A standalone SH had 2 multiple indexer clusters configured as per the docs below, which has been working well a while until I added 3rd cluster.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Indexer/Configuremulti-clustersearch

The searches appear working only across the existing 2 indexer clusters but the indexers from the 3rd cluster do not return any data.

[clustering]
mode = searchhead
master_uri = clustermaster:one, clustermaster:two, clustermaster:three

[clustermaster:three]
master_uri=https://SplunkMaster03.com:8089
pass4SymmKey=correctSecret

Please help.

Labels (2)
Tags (2)
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers.

The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results.

"Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue.

https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers.

The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results.

"Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue.

https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...