Monitoring Splunk

[Multiple Clusters] search not returning from new cluster

sylim_splunk
Splunk Employee
Splunk Employee

A standalone SH had 2 multiple indexer clusters configured as per the docs below, which has been working well a while until I added 3rd cluster.

https://docs.splunk.com/Documentation/Splunk/7.2.4/Indexer/Configuremulti-clustersearch

The searches appear working only across the existing 2 indexer clusters but the indexers from the 3rd cluster do not return any data.

[clustering]
mode = searchhead
master_uri = clustermaster:one, clustermaster:two, clustermaster:three

[clustermaster:three]
master_uri=https://SplunkMaster03.com:8089
pass4SymmKey=correctSecret

Please help.

Labels (2)
Tags (2)
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers.

The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results.

"Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue.

https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC

View solution in original post

sylim_splunk
Splunk Employee
Splunk Employee

Search.log in Search job inspector shows that searches were only dispatched to the 2 existing cluster indexers but not to the new indexers.

The search GUI shows the field name, splunk_server_group field shows that dmc_xxx groups. This means the deployment monitoring console, DMC is enabled on the search head. With DMC enabled on SH, only search peers that are members of the indexer group are searched by default. Because all searches that run on the monitoring console instance follow this behavior, non-monitoring console searches might have incomplete results.

"Do Not Use Production Search Head for DMC" which is also documented in the link below - Resolution: Switching to "standalone" next to "distributed" fixed the issue.

https://docs.splunk.com/Documentation/Splunk/7.2.4/DMC/WheretohostDMC

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...