You have a thread from 2020 that states they fixed their problem. I am pretty sure the reason the solution works is similar to what I am going to suggest here. I have found (no scientific evidence to support it) that sometimes the conf files just seem to be buggered and if reset them, it starts to work. I swear the settings are the same before the reset and after, but for some reason it works. Maybe it's voodoo or whatever, but it has worked for me in the past. Here is a breakdown of quickly resetting the configurations that you need The warning suggests the SH is trying to query a non-existent or misconfigured search peer, possibly due to stale or incorrect settings in outputs.conf or related configuration files. Resetting outputs.conf clears any corrupted or conflicting settings (e.g., incorrect server names, ports, or SSL configurations) that might be preventing the SH from recognizing the IDX as a valid peer. Restarting Splunk ensures a clean state, and re-adding the peer re-establishes the connection with fresh, verified settings. Steps to Reset and Reconfigure Back Up Configuration Files: Before making changes, back up your Splunk configuration files to avoid losing custom settings. On the Search Head, copy the $SPLUNK_HOME\etc\system\local directory (e.g., C:\Program Files\Splunk\etc\system\local) to a safe location (e.g., C:\SplunkBackup). Delete or Rename outputs.conf: Navigate to $SPLUNK_HOME\etc\system\local on the Search Head (e.g., C:\Program Files\Splunk\etc\system\local). Locate outputs.conf. If it exists, rename it to outputs.conf.bak (or delete it if you’re sure no critical settings are needed). Note: If outputs.conf is in an app directory (e.g., $SPLUNK_HOME\etc\apps\<app_name>\local), check there too and rename/delete it. This ensures Splunk starts with default output settings, clearing any misconfigurations. Restart Splunk on the Search Head: Open a Command Prompt as Administrator on the Windows SH host. Navigate to $SPLUNK_HOME\bin (e.g., cd "C:\Program Files\Splunk\bin"). Run: splunk restart This restarts the Splunk service, applying the reset configuration. Verify Indexer Configuration: Ensure the Indexer is configured to receive data on the correct port (default: 9997). On the Indexer, check $SPLUNK_HOME\etc\system\local\inputs.conf for a [splunktcp://9997] stanza: ini [splunktcp://9997]
disabled = 0 If missing, add it and restart the Indexer (splunk restart). Confirm port 9997 is open: netstat -an | findstr 9997 (should show LISTENING). Reconfigure the Search Peer: On the Search Head, log into the Splunk Web UI as an admin. Go to Settings > Distributed Search > Search Peers. Remove the existing Indexer peer (select the IDX and click Remove). Add the Indexer as a new peer: Click Add New. Enter the Indexer’s details: Peer URI: https://<Indexer_IP>:8089 (e.g., https://192.168.1.100:8089). Authentication: Use the SH admin credentials or a pass4SymmKey (if configured in distsearch.conf). Replication Settings: Ensure settings match your setup (usually default). Save and wait for the status to show Healthy. Alternatively, use the CLI: cmd splunk add search-server https://<Indexer_IP>:8089 -auth <admin>:<password> -remoteUsername <admin> -remotePassword <password> Test the Search: Run your search again from the SH: index=_internal. Verify results are returned without the warning. Check the Monitoring Console (Settings > Monitoring Console > Search > Distributed Search Health) to confirm the peer is active and responding. Additional Tips Check Network Connectivity: Ensure the SH can reach the IDX on port 8089 (management) and 9997 (data). Run: telnet <Indexer_IP> 8089 and telnet <Indexer_IP> 9997 from the SH host. If blocked, check Windows Firewall or network settings. Verify SSL Settings: If using SSL, ensure distsearch.conf on the SH and inputs.conf on the IDX align (e.g., ssl = true). Check $SPLUNK_HOME\var\log\splunk\splunkd.log on both hosts for SSL errors. Confirm Splunk Versions: Your SH and IDX should be on compatible versions (e.g., SH 8.2.2.1 or newer, IDX same or older). Run splunk version on both to confirm. If mismatched, upgrade the SH first. Debug Logs: If the issue persists, check $SPLUNK_HOME\var\log\splunk\splunkd.log
... View more