Search head returning no results from peer

[TAE2112]

I have a fairly basic deployment - one Search Head configured with two distributed search peers/indexers. Each peer is in a different data center, serving as the role of an indexer. I've installed the Universal Forwarder on the hosts in each datacenter, and they are successfully sending Windows Event Log data to their respective indexer. No clustering or replication is taking place, and both indexers have checked into the licensing service (which is also running on my Search Head). Both peers are showing up as "Up" and replication status is "Successful".

On each indexer, I can execute searches against indexes specific to the indexes hosted on each indexer. Put another way (because that sounded really confusing), when I search for data appropriate to each indexer, I receive the expected results.

While logged into the search head, when I search for data specific to data hosted by Indexer01, I receive the expected results. However, when I search for data hosted by Indexer02, I get no results. I've restarted the Splunk services many times, I've removed and re-added the "failing" peer, to no avail. Logs are "clean" in that I'm not finding any glaring errors in both the "splunkd" and "remote_searches" log files.

What am I missing? Can a search head only utilize a single peer? What log should I be looking at?

Thank you in advance!
-Todd

[anand_singh17]

Cause of this error:

• Cluster-bundle applied, but SH was busy, could not take up the update.

Resolution:
1. restart the splunkd service on SH.
2. Rejoin the SH to the cluster

[TAE2112 · Accepted solution]

Okay, the resolution to this was almost painful in its simplicity.

The fix:

1. On the search head, click Settings > Distributed Management Console
2. Note with frustration that only one Indexer is listed. Click "Setup" in the little green menu bar near the top.
3. See that both remote instances (Indexers) are listed, but the one which is failing is not listed as "Configured". (I think it was offline or something.)
4. On the Indexer which isn't working, under the "Actions" column, select Edit > Edit Server roles.
5. Make no changes to the list of roles. (Seriously - no changes.)
6. Click "Save"
7. Wait a moment, and then search for data specific to the second Indexer - the one which wasn't working.
8. Cheer when you get data back.

I'm not sure how I missed this in the documentation, but that's all it took. Hope this helps someone else.

-Todd

[knulps]

Thanks for this solution. it worked for me. I went ahead and made my Monitoring Console as standalone.  that fixed the issue. 

[Kieffer87]

Not sure I would have thought to do this since I don't use this search head as my DMC.

[divyamudundi]

This fixed the issue when our search head peer has problem communicating to one of the remote instance.

[RecoMark0]

This has worked for me multiple times, thank you. I seem to have an issue every time I restart my indexers, where no data comes in to the search head. I just "fake edit" the two indexers in the server roles column, and apply changes. After that everything works!

[jdaves]

Thank you. I've added "Spin through DMC and save configuration any time architecture is modified" to our checklist... ugh.

[bandit]

Splunk please fix this bug!!! I would have never figured this out without the above answer. I ran into this issue by removing a search peer then added it back and no longer got search results back from the search peer. The workaround above worked for me. Not sure why the DMC interacts with distsearch.conf config.

[phoffman_splunk]

Define your index in your search from search head index=

Also, verify that users roll has permissions to search that index (in Settings >> Access controls >> Roles)

You can verify that the distributed search is quering both peers by checking the job inspector (after search, click on the "job dropdown")

[TAE2112]

Thanks for the response!

Bad news: Unfortunately, it didn't have anything to do with roles

I thought you may have hit upon the piece I was missing - the "role permissions" suggestion - but it didn't change anything, as the request is still not returning data. Further, I've been using the working indexer as a point of reference (set up by someone else), and this server doesn't have anything special set up with regards to permissions and indexes.

The job inspector feature, however, does confirm the fact that only one of the two indexers (the one which is working) is being queried for information based on my search criteria. The other indexer is mysteriously absent, even though it shows as being "up" in the list of search peers.

I've wrangled some time with the individual who set up the original (working) indexer as a peer, so I hope I'll get an answer shortly.

[Raghav2384]

Is Indexer 2 and Universal Forwarder communicating? By your description, looks like indexer 1 is all good.

Check your outputs.conf and inputs.conf on the UF again. Can you launch splunkweb on the Indexer2 and see if it shows any events in the data summary wiz?

Thanks,
Raghav

[TAE2112]

Yep - all communication is successful. UF's on the servers are all reporting to their respective indexer (the indexer specific to that datacenter). The Search Head server and Indexers are both talking to one another. It's all very weird - if they can talk, why can't they exchange information?