Dedicated Monitoring Console configuration problem - "splunk_server/splunk_server_group do not match any search peer"

[ikulcsar]

Hi there,

I'm building a test Splunk deployment: 3 SH in cluster, 2x2 IX in multi-site cluster, 1 admin node(CM, Deployer, ...) and 1 dedicated Monitoring Console node. I have a problem with the Monitoring Console setup.
I tried to follow the documentation (https://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/Deploymentsetupsteps)

I've added as Search peer:
- all SH server
- admin node (incl. Cluster Master role)

I've enabled the Distributed Monitor Console, fixed instances' roles if needed. Apply.

Results:
- Under Overview->Topology there are no Indexers listed.
- There are several panels which are empty and have a warning: "Search filters specified using splunk_server/splunk_server_group do not match any search peer."

What am I doing wrong? Please help me fix it.

Regards,
István

[Wohamed_wakkad]

Steps of implementation MC at indexer cluster deployment infrastructure.

1-connect the Monitoring console to Cluster Master as a search head.

2-Forward all component internal logs to indexer (SH,LM,DS,CM) note: DS internal logs are stored locally and forwarded via selective forwarding

3-Set search head cluster and indexer cluster labels (normally configured when u established indexer cluster)

4- Add all instances as search peers (SH,LM,DS,CM) except indexers which are member of clusters.

5-Set up the monitoring console in distributed mode as bellow

1. Log into the instance on which you want to configure the monitoring console. The instance by default is in standalone mode, unconfigured.
2. In Splunk Web, select Monitoring Console > Settings > General Setup.
3. Click Distributed mode.
4. Confirm the following:
   • The columns labeled instance and machine are populated correctly and show unique values within each column.
   • The server roles are correct. For example, a search head that is also a license manager must have both server roles listed. If not, click Edit > Edit Server Roles and select the correct server roles for the instance.
   • If you are using indexer clustering, make sure the cluster manager instance is set to the cluster manager server role. If not, click Edit > Edit Server Roles and select the correct server role.
   • If you are hosting the monitoring console on an instance other than the cluster manager, you must add the cluster manager instance as a search peer and configure the monitoring console instance as a search head in that cluster. See Enable a search head in Managing Indexers and Clusters of Indexers.
   • To monitor a multisite indexer cluster, you must configure the monitoring console as a multisite search head. See Configure the search heads in Managing Indexers and Clusters of Indexers.
   • Make sure anything marked as an indexer is actually an indexer.
5. (Optional) Set custom groups. Custom groups are tags that map directly to distributed search groups. You might find groups useful, for example, if you have multisite indexer clustering in which each group can consist of the indexers in one location, or if you have an indexer cluster plus standalone peers. Custom groups are allowed to overlap. For example, one indexer can belong to multiple groups. See Create distributed search groups in the Distributed Search manual.
6. Click Apply Changes.

the above section configuration at the screenshot shared with the comment

don't use the bellow configuration :

1. Edit the splunk_monitoring_console_assets.conf file in etc/apps/splunk_monitoring_console/local.
2. Under the settings stanza, set mc_auto_config to enable, as shown:

                   [settings]

                   mc_auto_config = enabled

[aruncp333]

Does that mean your indexer cluster would have 4 search heads in SHC(as per your lab setup)? 

[zshy_splunk]

The answer is already provided but wanted to explain the logic of it.
There are two ways that a distributed search is configured. One for non-clustered Indexers and one for clustered Indexers.
The one for non-clustered Indexers is done via adding the Indexers as Search Peers, the other for clustered Indexers is done by adding the Search Head to the cluster via Indexer Clustering.
The Monitoring Console (MC) is using the non-clustered method to connect to all instances it is monitoring (Adding those as Search Peers). The documentation assumes the MC is already connected to the cluster via the Indexer Cluster settings so it is not required that the clustered Indexers be added as standalone Indexers (Search Peers).
The Cluster Master should be added as a Search Peer like the rest of the instances the MC monitors so it will be searchable as it is not searchable via the Indexer Cluster configuration.
In short, both configurations are required. The Cluster Master as a Search peer and the Monitoring Console as a Search Head in the Cluster.
Hope this clarifies the requirements for a standalone MC monitoring clustered Indexers.

[ikulcsar]

Hi,

Thanks for your help here as well.
Only one note:
I think the documentation shouldn't assume that MC is already connected to the cluster via the Indexer Cluster settings (not listed in the prerequisites list). Not even because docs say: do not add clustered indexers as a search peer. But connecting MC to the cluster via the Indexer Cluster settings adds all the indexer as a search peer. (Correct me if I'm wrong.)

So a little modification on the documentation would make this clear.

Regards,
István

[harsmarvania57]

Hi,

I am not sure why Doc is saying that http://docs.splunk.com/Documentation/Splunk/7.2.1/DMC/Addinstancesassearchpeers, you need to add Cluster Master as a search peer in MC. You need to point MC node to CM same as you pointed SHC members to CM to search data from Indexer Cluster (In my lab environment I have pointed MC to CM and it is automatically populating all Indexers in MC).

EDIT: I have submitted feedback on that documentation, let's see what Docs team will say.

[ikulcsar]

This Is what you are pointing to?: "Repeat these steps for each search head, deployment server, license master, and nonclustered indexer. Do not add clustered indexers, but be sure to add clustered search heads. If you are monitoring an indexer cluster and you are hosting the monitoring console on an instance other than the cluster master, you must add the cluster master as a search peer."

It says add cm as search peer

I also added the CM as Search peer to the MC node. MC also recognized it as a Cluster Master too.

[harsmarvania57]

Yes, instead of adding CM as search peer, can you please point MC node to CM same as SHC members points to CM to search data from Indexer Cluster Members (Ref. http://docs.splunk.com/Documentation/Splunk/7.2.1/DistSearch/SHCandindexercluster)

[ikulcsar]

Ohh, sorry, I misunderstand you.

I added MC as IX Cluster Search peer - IXs look good. But "Indexer Clustering: Status" page doesn't. I also add CM as Distributed Search peer. Now it looks good.
So now:
- MC is Cluster Search peer to the CM (it is added all the IX as Distributed Search peer)
- On the MC CM added as Distributed Search peer

Documentation does not say that at all. It looks like a support ticket will be opened...

Thx.

[harsmarvania57]

Yes, while double checking in my lab environment found that I have also added CM as search peer on MC.