Monitoring Splunk

Monitoring Console configuration problem||Indexers (which part of indexer cluster) does appear at MC||mc_auto_config

Wohamed_wakkad
Path Finder

"I am writing this thread to summarize the Monitoring Console (MC) configuration and highlight the non-obvious nuances within the configurations and troubleshooting/error sections."

Steps for Implementing the Monitoring Console (MC) in an Indexer Cluster Infrastructure

1. Connect the Monitoring Console to the Cluster Manager

Configure the Monitoring Console as a search head and connect it to the Cluster Manager (formerly Cluster Master).  // if  not configured the indexers doesn't shown at MC dashboard

[clustering]

manager_uri = https://Manager-URI:8089

mode = searchhead

pass4SymmKey = whatever

2. Forward Internal Logs to the Indexers

Ensure all component internal logs (_internal and _introspection) are forwarded to the indexer cluster from all roles (Search Heads, License Masters, Deployment Servers, and Cluster Managers).

Note: Deployment Server (DS) internal logs are stored locally and must be forwarded to the indexers using selective forwarding configuration.

[indexAndForward]
# Strictly disables local indexing on this host
index = false

[tcpout]
defaultGroup = static_indexer_cluster

[tcpout:static_indexer_cluster]
server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997

for DS:

[indexAndForward]
index = true
selectiveIndexing = true

[tcpout]
# Prevents automatic untargeted forwarding of every input
defaultGroup = none

[tcpout:static_indexer_cluster]
server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997

 

3. Set Search Head Cluster and Indexer Cluster Labels

Verify and apply distinct cluster labels for both your Search Head Clusters (SHC) and Indexer Clusters. (Note: These are typically defined during the initial cluster initialization).

4. Add All Component Instances as Search Peers

Add all non-indexer instances (Search Heads, License Masters, Deployment Servers, and Cluster Managers) to the Monitoring Console as search peers. Do not manually add individual indexers that are already members of the managed indexer cluster, as they are discovered automatically via the Cluster Manager.

from settings>Distributed search

search peer and then add them

5. Configure the Monitoring Console in Distributed Mode

Set up and enable the Monitoring Console in distributed mode as outlined below.

 from settings>Monitoring console

settings>general settings

ensure each machine has its role.

the search head roles are search head and KV store.

For more clarification about the above step look at the screenshots bellow:

Wohamed_wakkad_0-1781357126015.png

 

the final expected output:

Wohamed_wakkad_1-1781357233107.png

finally for the bellow section review @kml_uvce  comment

enable automatic distributed mode configuration:

  1. Edit the splunk_monitoring_console_assets.conf file in etc/apps/splunk_monitoring_console/local.
  2. Under the settings stanza, set mc_auto_config to enable, as shown:
     

    [settings]

    mc_auto_config = enabled

     

    because when check the configuration it cause an warning :                                                                                                                                                                                                                                                                                                 Invalid key in stanza [settings] in /opt/splunk/etc/apps/splunk_monitoring_console/local/splunk_monitoring_console_assets.conf, line 2: mc_auto_config (value: enabled).
Labels (1)
0 Karma

kml_uvce
Builder

On the last item, the mc_auto_config section is actually fine as written. The value enabled is correct — it's exactly what the official docs show:

 
 
[settings]mc_auto_config = enabled

https://help.splunk.com/en/splunk-enterprise/administer/monitor/10.4/configure-the-monitoring-consol...

The message you're hitting —

Invalid key in stanza [settings] … mc_auto_config (value: enabled)

— is a conf-validation warning, not a failure. That "Invalid key in stanza" wording fires when the key isn't found in the matching .conf.spec file. In several builds mc_auto_config isn't listed in splunk_monitoring_console_assets.conf.spec, so btool/conf validation flags it cosmetically even though splunkd honors it at runtime — same class as the old python.version warning that can be safely ignored.

So the config isn't broken. To confirm it's actually applying, don't go by the warning — check whether the MC auto-populates in distributed mode (Settings > General Setup, asset table). Per the docs, with mc_auto_config = enabled it takes effect on restart, or automatically every hour with no restart required.

kamal singh bisht
0 Karma

Wohamed_wakkad
Path Finder

thx i refer to your reply at the base topic

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

A Four-Part Event Series: Full Stack Observability For the AI Era

As AI reshapes applications, infrastructure, and the way teams operate, the traditional boundaries of ...

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...