After adding two new indexers as search peers to my existing search head, why is my search not returning results from an index on these instances?

[stevepraz]

I have an environment that has two indexers. I recently added an additional two indexers and added them as search peers to my existing search head. All 4 indexers have an index called "pcoip" that stores data related to virtual desktops.

When I run this search index=pcoip, I only get results from the original two indexers, even though all four have data in that index during the specified time frame.

When I run the search and add the splunk_server fieldindex=pcoip splunk_server=*, I get results back for all four indexers.

Is there some setting or configuration that I am missing that prevents these searches from returning the same data?

[maciep · Accepted solution]

What version of Splunk? Is DMC configured? There's a known issue we've run into related to DMC where some of our indexers don't get searched. The workaround is to just click apply on the set up page of DMC

SPL-99116
After enabling the Distributed Management Console (DMC) in distributed mode in an indexing cluster, the search head may not be able to search all the peers. The error will mention splunk_server_group : "Search filters specified using splunk_server/splunk_server_group do not match any search peer". To work around the issue, go to the DMC setup page and click Apply. To avoid the issue, run the DMC in standalone mode.

[MuS]

Yesterday I ran into this on a Splunk 6.3.0 instance....looks like this feature is still available 🙂

[stevepraz]

Not sure how to give maciep the credit but that hit the nail on the head.

I am using DMC and by going in to the Setup screen and hitting Apply, my search is now able to correctly pull results from all indexers.

[ppablo]

Hi @stevepraz

Just converted @maciep's comment under your question to an answer and accepted it 🙂 To give maciep even more credit, you can always upvote their answer so they get a boost of 15 karma points. Cheers!

Patrick

[stevepraz]

Currently running 6.2.1 on the search head and original indexers and 6.2.4 on the new indexers. I do have DMC configured.

When I went into DMC, I saw the two new indexers listed as State of "New". I hit apply changes. After that I ran the search again and it worked.

I never actually saw the error mentioned above but that fix appears to have worked.

[nvanderwalt_spl]

Did you edit distsearch.conf on your searchead to add the two servers in?

Check out http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/Configuredistributedsearch

[stevepraz]

Yes. I configured the indexers as Search Peers using Splunk web.

[woodcock]

I would open a support case (be sure to let us know what you find out).