Dedicated Monitoring Console configuration problem - "splunk_server/splunk_server_group do not match any search peer"

[ikulcsar]

Hi there,

I'm building a test Splunk deployment: 3 SH in cluster, 2x2 IX in multi-site cluster, 1 admin node(CM, Deployer, ...) and 1 dedicated Monitoring Console node. I have a problem with the Monitoring Console setup.
I tried to follow the documentation (https://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/Deploymentsetupsteps)

I've added as Search peer:
- all SH server
- admin node (incl. Cluster Master role)

I've enabled the Distributed Monitor Console, fixed instances' roles if needed. Apply.

Results:
- Under Overview->Topology there are no Indexers listed.
- There are several panels which are empty and have a warning: "Search filters specified using splunk_server/splunk_server_group do not match any search peer."

What am I doing wrong? Please help me fix it.

Regards,
István

[aruncp333]

Does that mean your indexer cluster would have 4 search heads in SHC(as per your lab setup)? 

[zshy_splunk]

The answer is already provided but wanted to explain the logic of it.
There are two ways that a distributed search is configured. One for non-clustered Indexers and one for clustered Indexers.
The one for non-clustered Indexers is done via adding the Indexers as Search Peers, the other for clustered Indexers is done by adding the Search Head to the cluster via Indexer Clustering.
The Monitoring Console (MC) is using the non-clustered method to connect to all instances it is monitoring (Adding those as Search Peers). The documentation assumes the MC is already connected to the cluster via the Indexer Cluster settings so it is not required that the clustered Indexers be added as standalone Indexers (Search Peers).
The Cluster Master should be added as a Search Peer like the rest of the instances the MC monitors so it will be searchable as it is not searchable via the Indexer Cluster configuration.
In short, both configurations are required. The Cluster Master as a Search peer and the Monitoring Console as a Search Head in the Cluster.
Hope this clarifies the requirements for a standalone MC monitoring clustered Indexers.

[ikulcsar]

Hi,

Thanks for your help here as well.
Only one note:
I think the documentation shouldn't assume that MC is already connected to the cluster via the Indexer Cluster settings (not listed in the prerequisites list). Not even because docs say: do not add clustered indexers as a search peer. But connecting MC to the cluster via the Indexer Cluster settings adds all the indexer as a search peer. (Correct me if I'm wrong.)

So a little modification on the documentation would make this clear.

Regards,
István

[harsmarvania57]

Hi,

I am not sure why Doc is saying that http://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/Addinstancesassearchpeers, you need to add Cluster Master as a search peer in MC. You need to point MC node to CM same as you pointed SHC members to CM to search data from Indexer Cluster (In my lab environment I have pointed MC to CM and it is automatically populating all Indexers in MC).

EDIT: I have submitted feedback on that documentation, let's see what Docs team will say.

[ikulcsar]

This Is what you are pointing to?: "Repeat these steps for each search head, deployment server, license master, and nonclustered indexer. Do not add clustered indexers, but be sure to add clustered search heads. If you are monitoring an indexer cluster and you are hosting the monitoring console on an instance other than the cluster master, you must add the cluster master as a search peer."

It says add cm as search peer

I also added the CM as Search peer to the MC node. MC also recognized it as a Cluster Master too.

[harsmarvania57]

Yes, instead of adding CM as search peer, can you please point MC node to CM same as SHC members points to CM to search data from Indexer Cluster Members (Ref. http://docs.splunk.com/Documentation/Splunk/7.2.1/DistSearch/SHCandindexercluster)

[ikulcsar]

Ohh, sorry, I misunderstand you.

I added MC as IX Cluster Search peer - IXs look good. But "Indexer Clustering: Status" page doesn't. I also add CM as Distributed Search peer. Now it looks good.
So now:
- MC is Cluster Search peer to the CM (it is added all the IX as Distributed Search peer)
- On the MC CM added as Distributed Search peer

Documentation does not say that at all. It looks like a support ticket will be opened...

Thx.

[harsmarvania57]

Yes, while double checking in my lab environment found that I have also added CM as search peer on MC.