We would like to inquire about a communication issue between the Search Head and the Indexers that occurred after the Indexer Cluster was restored. Environment Splunk Enterprise 9.3.3 Indexer Cluster consisting of three Indexers Search Head environment Background we experienced an issue where the Indexer Cluster did not recover properly after all Indexers had been shut down. Originally, indexes.conf was deployed directly to each Indexer. As part of our investigation and recovery efforts, we changed the configuration so that indexes.conf would be managed and deployed through the Cluster Manager. After this change, the Indexer Cluster recovered successfully. However, after the cluster itself returned to a healthy state, a new issue occurred in which search communication from the Search Head to the Indexers no longer functioned correctly. During troubleshooting, we temporarily interrupted the TCP/9997 connections on the Indexer side and then re-established them. After doing so, communication between the Search Head and the Indexers recovered successfully. Questions 1. ★ High priority Are there any known issues in which communication from a Search Head to the Indexers does not re-establish properly after an Indexer Cluster has recovered? 2. Since the issue was resolved after the TCP/9997 connections were re-established, could stale connection information, session inconsistencies, or cached communication states have contributed to the issue? 3. ★ High priority After all Indexers in an Indexer Cluster have been shut down and subsequently recovered, are there any additional steps or recommended procedures that should be performed on the Search Head, Deployer, or Cluster Manager? 4. Based on the information provided, are there any likely root causes or additional areas that Splunk Support would recommend investigating? ... View more

We would like to confirm the recommended management method for indexes.conf in an Indexer Cluster environment, as well as any behavioral changes in Splunk Enterprise 9.3.3. Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Background For clustered indexes, we have defined the following setting in indexes.conf: repFactor = auto   Our understanding is that, in an Indexer Cluster environment, it is recommended to manage indexes.conf under the Cluster Manager and distribute it to the Indexers through the Cluster Bundle. However, indexes.conf had been deployed directly on each Indexer since the initial implementation. This configuration operated without any issues in previous versions. After upgrading to Splunk Enterprise 9.3.3, we encountered an issue in which the Indexer Cluster failed to recover properly after all Indexers had been shut down. Questions Starting with Splunk Enterprise 9.3.x, have there been any enhancements to consistency validation checks or any behavioral changes regarding the management method or deployment location of indexes.conf and the repFactor=auto setting? Could deploying indexes.conf directly on each Indexer, rather than managing it through the Cluster Manager and Cluster Bundle, be a possible cause of the cluster recovery failure? ... View more

We would like to confirm the behavior of Universal Forwarders (UF) when all Indexers in an Indexer Cluster are stopped. Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Universal Forwarder Questions During a complete shutdown of all Indexers, are there any limitations on the amount of data or retention period for data buffered on the Universal Forwarders? ★ If all Indexers remain unavailable for approximately three weeks, is there a risk that the Universal Forwarders may be unable to retain all pending data, resulting in data loss? ★ After the Indexers are restored, what is the recommended procedure for resending any queued or unsent data from the Universal Forwarders? Are there any recommended best practices or maintenance procedures when shutting down and subsequently restarting all Indexers in an Indexer Cluster? The questions marked with ★ are our highest-priority items. We would greatly appreciate your guidance on those points first, if possible. ... View more