If I have a Splunk deployment with two Search Heads (one dedicated to Enterprise Security and one for core Splunk) and an Indexer Cluster, is it correct to install Splunk_TA_ForIndexers on the Indexer Cluster according to the approach below and if they are any recommendations? CM configurations Hierarchy: etc/manager-apps/ ├── 000_production_storage/ <-- Defines your global paths, volumes, and retention(ooo for precedence) ├── Splunk_TA_ForIndexers/ <-- Defines ES index names and unique security parsing └── Splunk_TA_windows/ <-- Shared standalone TA for both Search Heads   1-For 000_production_storage : A-Create a Base App on the Cluster Manager: Create a directory inside /opt/splunk/etc/manager-apps/ named 000_production_storage. B-Configure indexes.conf inside this app: Define your global storage volume paths (e.g., SSD for Hot, HDD for Cold) and explicitly map your enterprise security indexes (notable, risk, etc.) here to set their retention periods (frozenTimePeriodInSecs) and maximum size paths.   2-For shared TA that are used by both your ES and non-ES Search Heads (e.g., Splunk_TA_windows or Splunk_TA_fortinet_fortigate): A-Isolate the TA: Place the complete, standalone TA folder directly into the Cluster Manager's manager-apps directory. B-Exclude from ES Generation: * Log into your ES Search Head. Navigate to Enterprise Security > Configure > General >Distributed Configuration Management (DCM). Locate the shared TAs in the list and toggle them to Exclude. This prevents the DCM tool from duplicating these parsing configurations into the master bundle   3-For  Splunk_TA_ForIndexers  generate the stripped-down security configuration package for your indexers. A-Download the Package: In the same DCM menu on the ES Search Head, click Download to grab the auto-generated Splunk_TA_ForIndexers.spl file. B-Move to Staging & Extract: Transfer the .spl file to a staging directory or your Git pipeline and unpack it. C-Fix Cluster Replication (indexes.conf): Open Splunk_TA_ForIndexers/default/indexes.conf and globally change all instances of repFactor = 0 to:  repFactor = auto     4-Sanitize Paths (Best Practice): To prevent alphabetical conflicts with your storage layer, open indexes.conf inside this package and strip out the generic homePath, coldPath, and thawedPath lines from the stanzas. Leave only the index names and the repFactor = auto setting. This guarantees your 000_production_storage definitions win.   ... View more

"I am writing this thread to summarize the Monitoring Console (MC) configuration and highlight the non-obvious nuances within the configurations and troubleshooting/error sections." Steps for Implementing the Monitoring Console (MC) in an Indexer Cluster Infrastructure 1. Connect the Monitoring Console to the Cluster Manager Configure the Monitoring Console as a search head and connect it to the Cluster Manager (formerly Cluster Master).  // if  not configured the indexers doesn't shown at MC dashboard [clustering] manager_uri = https://Manager-URI:8089 mode = searchhead pass4SymmKey = whatever 2. Forward Internal Logs to the Indexers Ensure all component internal logs (_internal and _introspection) are forwarded to the indexer cluster from all roles (Search Heads, License Masters, Deployment Servers, and Cluster Managers). Note: Deployment Server (DS) internal logs are stored locally and must be forwarded to the indexers using selective forwarding configuration. [indexAndForward] # Strictly disables local indexing on this host index = false [tcpout] defaultGroup = static_indexer_cluster [tcpout:static_indexer_cluster] server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997 for DS: [indexAndForward] index = true selectiveIndexing = true [tcpout] # Prevents automatic untargeted forwarding of every input defaultGroup = none [tcpout:static_indexer_cluster] server = <indexer_01_ip>:9997, <indexer_02_ip>:9997, <indexer_03_ip>:9997   3. Set Search Head Cluster and Indexer Cluster Labels Verify and apply distinct cluster labels for both your Search Head Clusters (SHC) and Indexer Clusters. (Note: These are typically defined during the initial cluster initialization). 4. Add All Component Instances as Search Peers Add all non-indexer instances (Search Heads, License Masters, Deployment Servers, and Cluster Managers) to the Monitoring Console as search peers. Do not manually add individual indexers that are already members of the managed indexer cluster, as they are discovered automatically via the Cluster Manager. from settings>Distributed search search peer and then add them 5. Configure the Monitoring Console in Distributed Mode Set up and enable the Monitoring Console in distributed mode as outlined below.  from settings>Monitoring console settings>general settings ensure each machine has its role. the search head roles are search head and KV store. For more clarification about the above step look at the screenshots bellow:   the final expected output: finally for the bellow section review @kml_uvce  comment enable automatic distributed mode configuration: Edit the splunk_monitoring_console_assets.conf file in etc/apps/splunk_monitoring_console/local. Under the settings stanza, set mc_auto_config to enable, as shown:   [settings] mc_auto_config = enabled   because when check the configuration it cause an warning :                                                                                                                                                                                                                                                                                                 Invalid key in stanza [settings] in /opt/splunk/etc/apps/splunk_monitoring_console/local/splunk_monitoring_console_assets.conf, line 2: mc_auto_config (value: enabled). ... View more

According to Splunk Validated architecture of designing HA between 2 syslog server   the documentation says this -->  Using a network load-balancer (L4/L7) to provide a highly available syslog collection infrastructure is discouraged due to potential data loss and fidelity concerns (IP lost in proxy). The recommended approach is to use a cluster resource manager solution (e.g. pacemaker/keepalived) or a VM-based syslog server deployment instead. Splunk does not have any insight into available technologies at your company, so please engage the in-house experts in your network team to design and deploy such an HA solution. did anyone try this or have another common practice approach?   ... View more