Splunk Search

After adding two new indexers as search peers to my existing search head, why is my search not returning results from an index on these instances?

stevepraz
Path Finder

I have an environment that has two indexers. I recently added an additional two indexers and added them as search peers to my existing search head. All 4 indexers have an index called "pcoip" that stores data related to virtual desktops.

When I run this search index=pcoip, I only get results from the original two indexers, even though all four have data in that index during the specified time frame.

When I run the search and add the splunk_server fieldindex=pcoip splunk_server=*, I get results back for all four indexers.

Is there some setting or configuration that I am missing that prevents these searches from returning the same data?

0 Karma
1 Solution

maciep
Champion

What version of Splunk? Is DMC configured? There's a known issue we've run into related to DMC where some of our indexers don't get searched. The workaround is to just click apply on the set up page of DMC

SPL-99116
After enabling the Distributed Management Console (DMC) in distributed mode in an indexing cluster, the search head may not be able to search all the peers. The error will mention splunk_server_group : "Search filters specified using splunk_server/splunk_server_group do not match any search peer". To work around the issue, go to the DMC setup page and click Apply. To avoid the issue, run the DMC in standalone mode.

View solution in original post

maciep
Champion

What version of Splunk? Is DMC configured? There's a known issue we've run into related to DMC where some of our indexers don't get searched. The workaround is to just click apply on the set up page of DMC

SPL-99116
After enabling the Distributed Management Console (DMC) in distributed mode in an indexing cluster, the search head may not be able to search all the peers. The error will mention splunk_server_group : "Search filters specified using splunk_server/splunk_server_group do not match any search peer". To work around the issue, go to the DMC setup page and click Apply. To avoid the issue, run the DMC in standalone mode.

MuS
SplunkTrust
SplunkTrust

Yesterday I ran into this on a Splunk 6.3.0 instance....looks like this feature is still available 🙂

0 Karma

stevepraz
Path Finder

Not sure how to give maciep the credit but that hit the nail on the head.

I am using DMC and by going in to the Setup screen and hitting Apply, my search is now able to correctly pull results from all indexers.

0 Karma

ppablo
Retired

Hi @stevepraz

Just converted @maciep's comment under your question to an answer and accepted it 🙂 To give maciep even more credit, you can always upvote their answer so they get a boost of 15 karma points. Cheers!

Patrick

0 Karma

stevepraz
Path Finder

Currently running 6.2.1 on the search head and original indexers and 6.2.4 on the new indexers. I do have DMC configured.

When I went into DMC, I saw the two new indexers listed as State of "New". I hit apply changes. After that I ran the search again and it worked.

I never actually saw the error mentioned above but that fix appears to have worked.

nvanderwalt_spl
Splunk Employee
Splunk Employee

Did you edit distsearch.conf on your searchead to add the two servers in?

Check out http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/Configuredistributedsearch

0 Karma

stevepraz
Path Finder

Yes. I configured the indexers as Search Peers using Splunk web.

0 Karma

woodcock
Esteemed Legend

I would open a support case (be sure to let us know what you find out).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...