Warning when searching without results

[hendriks]

Hello, 

I have a simple distributed search config on a windows host, 1 SH, 1 IDX and 1 License server. Running a search from the SH give me a warning  : "Search filters specified using splunk_server/splunk_server_group do not match any search peer." And the search does not return any results.  (searching for index=_internal)

The answers found on this same topic over here do not seem to solve the problem for me. 

I recreated the user and the role, no success, I recreated the search peer, without success. 

Status under distributed search is healthy and replication status is Successful. 

Any suggestions what i could do to get distributed search up and running?

Richard

[anilchaithu]

@hendriks 

I am assuming you are running the search on SH.

Are you forwarding all the data from SH to Indexer?

Did you add search peer on search head? settings -> Distributed search -> search peers -> add new (you should add your indexer here)

Hope this helps

[hendriks]

@anilchaithu  thank  you for your reply, all you suggested i did. 

Yes i did run the search from the SH, thats where i see the warning, in the search Job inspector:

The following messages were returned by the search subsystem:

• warn : Search filters specified using splunk_server/splunk_server_group do not match any search peer.

 I'm forwarding all logs to the indexer and can see/search for them there (index=_internal host=shserver.local) and get results. 

I added the indexer as searchpeer,  when looking in  myserver:8000/en-GB/manager/splunk_monitoring_console/search/distributed/peers for this server (the only one in the list) the state is up,  health status is healthy and the replication status Successful, cluster label is none.   

Richard

[anilchaithu]

@hendriks 

Please check search.log for final search distributed to search peers. It will indicate the final search (along with search filters)

my guess is the splunk_server search filter is not matching with the indexer. 

[hendriks]

thanks again, I don't see the field you are referring to.  But I see there are some fields missing when comparing the log from another a server that succeeds the distuributed search and this where it fails. 

searchProviders on the one it works, contains the name of the searchpeer, on my server where it fails it contains the name of the server itself. 

Field that are missing : remoteSearchLogs, peerNameList, they show the name of the searchpeer on the server it works, the fields are missing on the sever where it does not work. 

Richard

[hendriks · Accepted solution]

@anilchaithu thanks, I was able to solve it now. I made the server standalone again, so removed forwarding of logs, removed the search pear. Doing a search still gave the same problem so i decided to  add the indexserver role. After the restart the localsearch  worked. It gave results on index _interal. After this i added back the Searchpeer, forwarding of logs and as last I removed the Indexer role.  A restart later and all still worked.  So don't know what really was wrong but I think some pff the configs was wonkie somehow.. 

Reverting the searchhead to an almost standalone server and back to a distributed searchhead fixed it in the end. 

Hope this will help others who run in this unclear to solve this issue. 

Richard

[LAME-Creations]

You have a thread from 2020 that states they fixed their problem.  I am pretty sure the reason the solution works is similar to what I am going to suggest here.  I have found (no scientific evidence to support it) that sometimes the conf files just seem to be buggered and if reset them, it starts to work.  I swear the settings are the same before the reset and after, but for some reason it works.  Maybe it's voodoo or whatever, but it has worked for me in the past.

Here is a breakdown of quickly resetting the configurations that you need

The warning suggests the SH is trying to query a non-existent or misconfigured search peer, possibly due to stale or incorrect settings in outputs.conf or related configuration files. Resetting outputs.conf clears any corrupted or conflicting settings (e.g., incorrect server names, ports, or SSL configurations) that might be preventing the SH from recognizing the IDX as a valid peer. Restarting Splunk ensures a clean state, and re-adding the peer re-establishes the connection with fresh, verified settings.

Steps to Reset and Reconfigure

1. Back Up Configuration Files:
   • Before making changes, back up your Splunk configuration files to avoid losing custom settings.
   • On the Search Head, copy the $SPLUNK_HOME\etc\system\local directory (e.g., C:\Program Files\Splunk\etc\system\local) to a safe location (e.g., C:\SplunkBackup).
2. Delete or Rename outputs.conf:
   • Navigate to $SPLUNK_HOME\etc\system\local on the Search Head (e.g., C:\Program Files\Splunk\etc\system\local).
   • Locate outputs.conf. If it exists, rename it to outputs.conf.bak (or delete it if you’re sure no critical settings are needed).
   • Note: If outputs.conf is in an app directory (e.g., $SPLUNK_HOME\etc\apps\<app_name>\local), check there too and rename/delete it.
   • This ensures Splunk starts with default output settings, clearing any misconfigurations.
3. Restart Splunk on the Search Head:
   • Open a Command Prompt as Administrator on the Windows SH host.
   • Navigate to $SPLUNK_HOME\bin (e.g., cd "C:\Program Files\Splunk\bin").
   • Run: splunk restart
   • This restarts the Splunk service, applying the reset configuration.
4. Verify Indexer Configuration:
   • Ensure the Indexer is configured to receive data on the correct port (default: 9997).
   • On the Indexer, check $SPLUNK_HOME\etc\system\local\inputs.conf for a [splunktcp://9997] stanza:
     ini

     [splunktcp://9997]
     disabled = 0

   • If missing, add it and restart the Indexer (splunk restart).
   • Confirm port 9997 is open: netstat -an | findstr 9997 (should show LISTENING).
5. Reconfigure the Search Peer:
   • On the Search Head, log into the Splunk Web UI as an admin.
   • Go to Settings > Distributed Search > Search Peers.
   • Remove the existing Indexer peer (select the IDX and click Remove).
   • Add the Indexer as a new peer:
     • Click Add New.
     • Enter the Indexer’s details:
       • Peer URI: https://<Indexer_IP>:8089 (e.g., https://192.168.1.100:8089).
       • Authentication: Use the SH admin credentials or a pass4SymmKey (if configured in distsearch.conf).
       • Replication Settings: Ensure settings match your setup (usually default).
     • Save and wait for the status to show Healthy.
   • Alternatively, use the CLI:
     cmd

     splunk add search-server https://<Indexer_IP>:8089 -auth <admin>:<password> -remoteUsername <admin> -remotePassword <password>

6. Test the Search:
   • Run your search again from the SH: index=_internal.
   • Verify results are returned without the warning.
   • Check the Monitoring Console (Settings > Monitoring Console > Search > Distributed Search Health) to confirm the peer is active and responding.

Additional Tips

• Check Network Connectivity: Ensure the SH can reach the IDX on port 8089 (management) and 9997 (data). Run: telnet <Indexer_IP> 8089 and telnet <Indexer_IP> 9997 from the SH host. If blocked, check Windows Firewall or network settings.
• Verify SSL Settings: If using SSL, ensure distsearch.conf on the SH and inputs.conf on the IDX align (e.g., ssl = true). Check $SPLUNK_HOME\var\log\splunk\splunkd.log on both hosts for SSL errors.
• Confirm Splunk Versions: Your SH and IDX should be on compatible versions (e.g., SH 8.2.2.1 or newer, IDX same or older). Run splunk version on both to confirm. If mismatched, upgrade the SH first.
• Debug Logs: If the issue persists, check $SPLUNK_HOME\var\log\splunk\splunkd.log

[_olivier_]

Hi, @hendriks ,  this is an old post, but can you remember the actions to add the indexserver role ? 

Thanks.

[hendriks]

Hi _olivier_,

Yes, off course when on your server go to the monitoring console, there under the menu setting, select "general setup" and there you can set the server roles. 

Kind regards. 

[gcusello]

Hi @_olivier_ ,

don't attach a new question on an old one, even if on the same topic: open a new request, so you will be more sure to receive an answer.

Ciao.

Giuseppe