Warning when searching without results

[hendriks]

Hello, 

I have a simple distributed search config on a windows host, 1 SH, 1 IDX and 1 License server. Running a search from the SH give me a warning  : "Search filters specified using splunk_server/splunk_server_group do not match any search peer." And the search does not return any results.  (searching for index=_internal)

The answers found on this same topic over here do not seem to solve the problem for me. 

I recreated the user and the role, no success, I recreated the search peer, without success. 

Status under distributed search is healthy and replication status is Successful. 

Any suggestions what i could do to get distributed search up and running?

Richard

[anilchaithu]

@hendriks 

I am assuming you are running the search on SH.

Are you forwarding all the data from SH to Indexer?

Did you add search peer on search head? settings -> Distributed search -> search peers -> add new (you should add your indexer here)

Hope this helps

[hendriks]

@anilchaithu  thank  you for your reply, all you suggested i did. 

Yes i did run the search from the SH, thats where i see the warning, in the search Job inspector:

The following messages were returned by the search subsystem:

• warn : Search filters specified using splunk_server/splunk_server_group do not match any search peer.

 I'm forwarding all logs to the indexer and can see/search for them there (index=_internal host=shserver.local) and get results. 

I added the indexer as searchpeer,  when looking in  myserver:8000/en-GB/manager/splunk_monitoring_console/search/distributed/peers for this server (the only one in the list) the state is up,  health status is healthy and the replication status Successful, cluster label is none.   

Richard

[anilchaithu]

@hendriks 

Please check search.log for final search distributed to search peers. It will indicate the final search (along with search filters)

my guess is the splunk_server search filter is not matching with the indexer. 

[hendriks]

thanks again, I don't see the field you are referring to.  But I see there are some fields missing when comparing the log from another a server that succeeds the distuributed search and this where it fails. 

searchProviders on the one it works, contains the name of the searchpeer, on my server where it fails it contains the name of the server itself. 

Field that are missing : remoteSearchLogs, peerNameList, they show the name of the searchpeer on the server it works, the fields are missing on the sever where it does not work. 

Richard

[hendriks · Accepted solution]

@anilchaithu thanks, I was able to solve it now. I made the server standalone again, so removed forwarding of logs, removed the search pear. Doing a search still gave the same problem so i decided to  add the indexserver role. After the restart the localsearch  worked. It gave results on index _interal. After this i added back the Searchpeer, forwarding of logs and as last I removed the Indexer role.  A restart later and all still worked.  So don't know what really was wrong but I think some pff the configs was wonkie somehow.. 

Reverting the searchhead to an almost standalone server and back to a distributed searchhead fixed it in the end. 

Hope this will help others who run in this unclear to solve this issue. 

Richard