Splunk Search

Unable to run any search query : WARN: Search filters specified using splunk_server/splunk_server_group do not match any search peer.

splunker12er
Motivator

WARN: Search filters specified using splunk_server/splunk_server_group do not match any search peer.

Possibilities :
relax the primary search criteria -> (index=* doesnt work)
widen the time range of the search ->(time range chosen in 'all time')
check that the default search indexes for your account include the desired indexes -> (admin role -> using default settings)

what could be the cause ?

Splunk version: Splunk 6.0.4 (build 207768)
Role : License master servers
Slaves version: Splunk 6.2.1 (build 245427)

Labels (1)
Tags (2)

openpath_llc
Explorer

Encountered this same bug on Splunk 8.0.2.1. The steps from @ii_splunk worked well for me also.

marcoscala
Builder

Same bug on 8.0.8. The workaround proposed worked!!!

0 Karma

terminaloutcome
Path Finder

Same, on 8.0.1.

0 Karma

ii_splunk
Path Finder

I think this is a bug that Splunk needs to fix.... here is the work around in case anyone gets this:

On your search head do the following:

Settings->Distributed Management Console
(NOTE: Indexers will have N/A shown)
Setup->Apply Changes->Refresh
(NOTE: No changes were actually made)

Verify fix by clicking "Overview" in Distributed Management Console; Indexers will now show correct indexing rate.

Search as normal; workaround complete.

lycollicott
Motivator

ii_splunk,
Why and how does that work? It worked for me, but I don't understand it at all.

Settings->Distributed Management
Console (NOTE: Indexers will have N/A
shown) Setup->Apply Changes->Refresh
(NOTE: No changes were actually made)

Verify fix by clicking "Overview" in
Distributed Management Console;
Indexers will now show correct
indexing rate.

0 Karma

triest
Communicator

Of particular note is that this affected all searches.

As far as I know no changes where made to our DMC setup; we noticed that all searches quit working on our cluster master with the above mentioned error message.

0 Karma

yannK
Splunk Employee
Splunk Employee

Here is the known bug SPL-99116

After enabling the Distributed Management Console DMC, in "distributed mode", in an indexing cluster, the search-head may not be able to search all the peers. The error will mention splunk_server_group : "Search filters specified using splunk_server/splunk_server_group do not match any search peer". The workarounds are to go to the DMC setup page and hit "apply". To avoid the issue switch the DMC to "single instance" mode.

http://docs.splunk.com/Documentation/Splunk/6.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_se...

MuS
Legend

Hi ii_splunk & kylekoza,

please file a bug report with Splunk Support if this is re-producable http://docs.splunk.com/Documentation/Splunk/6.2.0/Troubleshooting/HowtofileagreatSupportcase
But to be honest - I believe you had some trouble - this question is not related to Distributed management console. DMC is only available since Splunk 6.2 http://docs.splunk.com/Documentation/Splunk/6.2.0/ReleaseNotes/MeetSplunk#Distributed_management_con... and @splunker12er is using Splunk 6.0.4

cheers, MuS

0 Karma

ii_splunk
Path Finder

I can't reproduce at will but when the cluster get's in this "odd" state; I happened onto this work around. Has reoccured a few times on our cluster.

0 Karma

kylekoza
Explorer

I had the same issue and this fixed it. Thanks!

0 Karma

ridwanahmed
Path Finder

thank you! I had the same ridiculous issue haha

0 Karma

Lucas_K
Motivator

try putting splunk_server=* into your base search.

I just encountered this on a hunk install.

MuS
Legend

Hi splunker12er,

It is I again 😉

Does your License master, where you run this search, have any search peers configured? Check in the UI

http[s]://YourSplunkHostName:YourSplunkPort/en-GB/manager/search/search/distributed/peers

or by using this REST command on the license master:

| REST /services/search/distributed/peers

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...