We would like to inquire about a communication issue between the Search Head and the Indexers that occurred after the Indexer Cluster was restored. Environment Splunk Enterprise 9.3.3 Indexer Cluster consisting of three Indexers Search Head environment Background we experienced an issue where the Indexer Cluster did not recover properly after all Indexers had been shut down. Originally, indexes.conf was deployed directly to each Indexer. As part of our investigation and recovery efforts, we changed the configuration so that indexes.conf would be managed and deployed through the Cluster Manager. After this change, the Indexer Cluster recovered successfully. However, after the cluster itself returned to a healthy state, a new issue occurred in which search communication from the Search Head to the Indexers no longer functioned correctly. During troubleshooting, we temporarily interrupted the TCP/9997 connections on the Indexer side and then re-established them. After doing so, communication between the Search Head and the Indexers recovered successfully. Questions 1. ★ High priority Are there any known issues in which communication from a Search Head to the Indexers does not re-establish properly after an Indexer Cluster has recovered? 2. Since the issue was resolved after the TCP/9997 connections were re-established, could stale connection information, session inconsistencies, or cached communication states have contributed to the issue? 3. ★ High priority After all Indexers in an Indexer Cluster have been shut down and subsequently recovered, are there any additional steps or recommended procedures that should be performed on the Search Head, Deployer, or Cluster Manager? 4. Based on the information provided, are there any likely root causes or additional areas that Splunk Support would recommend investigating? ... View more

We would like to inquire about a communication issue between the Search Head and Indexers that occurred after cluster recovery. Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Search Head environment Background In our environment, we experienced an issue where the Indexer Cluster did not recover properly after all Indexers had been shut down. Originally, indexes.conf had been deployed directly on each Indexer. As part of our investigation and remediation efforts, we changed the configuration so that indexes.conf would be managed under the Cluster Manager, which successfully restored the Indexer Cluster. After the cluster itself returned to a healthy state, a new issue occurred in which search communications from the Search Head to the Indexers were no longer functioning correctly. During troubleshooting, we temporarily interrupted connectivity on TCP/9997 at the Indexer side and then re-established the connection. After doing so, communication between the Search Head and Indexers recovered successfully. Questions ① Are there any known issues where Search Head-to-Indexer communication fails to re-establish properly after Indexer Cluster recovery? ★ ② Since the issue was resolved after reconnecting TCP/9997, could stale connection information, session inconsistencies, or cached communication states have been a contributing factor? ③ Following a full shutdown and recovery of all Indexers in an Indexer Cluster, are there any additional steps or recommended procedures that should be performed on the Search Head, Deployer, or Cluster Manager? ★ ④ Based on the information provided, are there any likely root causes that Splunk Support would recommend investigating further? Questions marked with a ★ are high-priority items. If possible, we would appreciate it if you could address these points first. ... View more

We would like to confirm the recommended management method for indexes.conf in an Indexer Cluster environment, as well as any behavioral changes in Splunk Enterprise 9.3.3. Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Background For clustered indexes, we have defined the following setting in indexes.conf: repFactor = auto   Our understanding is that, in an Indexer Cluster environment, it is recommended to manage indexes.conf under the Cluster Manager and distribute it to the Indexers through the Cluster Bundle. However, indexes.conf had been deployed directly on each Indexer since the initial implementation. This configuration operated without any issues in previous versions. After upgrading to Splunk Enterprise 9.3.3, we encountered an issue in which the Indexer Cluster failed to recover properly after all Indexers had been shut down. Questions Starting with Splunk Enterprise 9.3.x, have there been any enhancements to consistency validation checks or any behavioral changes regarding the management method or deployment location of indexes.conf and the repFactor=auto setting? Could deploying indexes.conf directly on each Indexer, rather than managing it through the Cluster Manager and Cluster Bundle, be a possible cause of the cluster recovery failure? ... View more

We would like to confirm the behavior of Universal Forwarders (UF) when all Indexers in an Indexer Cluster are stopped. Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Universal Forwarder Questions During a complete shutdown of all Indexers, are there any limitations on the amount of data or retention period for data buffered on the Universal Forwarders? ★ If all Indexers remain unavailable for approximately three weeks, is there a risk that the Universal Forwarders may be unable to retain all pending data, resulting in data loss? ★ After the Indexers are restored, what is the recommended procedure for resending any queued or unsent data from the Universal Forwarders? Are there any recommended best practices or maintenance procedures when shutting down and subsequently restarting all Indexers in an Indexer Cluster? The questions marked with ★ are our highest-priority items. We would greatly appreciate your guidance on those points first, if possible. ... View more

We would like to confirm the behavior of Universal Forwarders (UFs) when all Indexers in an Indexer Cluster are stopped.   Environment Splunk Enterprise 9.3.3 Indexer Cluster (3 Indexers) Universal Forwarders   Questions ①During a complete shutdown of all Indexers, are there any limitations on the amount of data or retention period for data buffered on the Universal Forwarders? ★ ②If all Indexers remain unavailable for approximately three weeks, is there a risk that the Universal Forwarders may be unable to retain all pending data, resulting in data loss? ★ ③After the Indexers are restored, what is the recommended procedure for resending any queued or unsent data from the Universal Forwarders? ④Are there any recommended best practices or maintenance procedures when shutting down and subsequently restarting all Indexers in an Indexer Cluster?   Questions marked with a ★ are priorities for our company. If possible, we would appreciate it if you could address these points first. ... View more