"I am writing this thread to summarize the Monitoring Console (MC) configuration and highlight the non-obvious nuances within the configurations and troubleshooting/error sections."
Configure the Monitoring Console as a search head and connect it to the Cluster Manager (formerly Cluster Master). // if not configured the indexers doesn't shown at MC dashboard
[clustering]
manager_uri = https://Manager-URI:8089
mode = searchhead
pass4SymmKey = whatever
Ensure all component internal logs (_internal and _introspection) are forwarded to the indexer cluster from all roles (Search Heads, License Masters, Deployment Servers, and Cluster Managers).
Note: Deployment Server (DS) internal logs are stored locally and must be forwarded to the indexers using selective forwarding configuration.
[indexAndForward]
# Strictly disables local indexing on this host
index = false
[tcpout]
defaultGroup = static_indexer_cluster
[tcpout:static_indexer_cluster]
server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997
for DS:
[indexAndForward]
index = true
selectiveIndexing = true
[tcpout]
# Prevents automatic untargeted forwarding of every input
defaultGroup = none
[tcpout:static_indexer_cluster]
server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997
Verify and apply distinct cluster labels for both your Search Head Clusters (SHC) and Indexer Clusters. (Note: These are typically defined during the initial cluster initialization).
Add all non-indexer instances (Search Heads, License Masters, Deployment Servers, and Cluster Managers) to the Monitoring Console as search peers. Do not manually add individual indexers that are already members of the managed indexer cluster, as they are discovered automatically via the Cluster Manager.
from settings>Distributed search
search peer and then add them
Set up and enable the Monitoring Console in distributed mode as outlined below.
from settings>Monitoring console
settings>general settings
ensure each machine has its role.
the search head roles are search head and KV store.
For more clarification about the above step look at the screenshots bellow:
the final expected output:
finally for the bellow section review @kml_uvce comment
enable automatic distributed mode configuration:
[settings]
mc_auto_config = enabled
because when check the configuration it cause an warning : Invalid key in stanza [settings] in /opt/splunk/etc/apps/splunk_monitoring_console/local/splunk_monitoring_console_assets.conf, line 2: mc_auto_config (value: enabled).
On the last item, the mc_auto_config section is actually fine as written. The value enabled is correct — it's exactly what the official docs show:
[settings]mc_auto_config = enabled
The message you're hitting —
Invalid key in stanza [settings] … mc_auto_config (value: enabled)
— is a conf-validation warning, not a failure. That "Invalid key in stanza" wording fires when the key isn't found in the matching .conf.spec file. In several builds mc_auto_config isn't listed in splunk_monitoring_console_assets.conf.spec, so btool/conf validation flags it cosmetically even though splunkd honors it at runtime — same class as the old python.version warning that can be safely ignored.
So the config isn't broken. To confirm it's actually applying, don't go by the warning — check whether the MC auto-populates in distributed mode (Settings > General Setup, asset table). Per the docs, with mc_auto_config = enabled it takes effect on restart, or automatically every hour with no restart required.
thx i refer to your reply at the base topic