Deployment Architecture

One Distributed Search Head Constantly Returning No Results

gstefancyk
Path Finder

I have two separate search heads, one for admins to use and another for regular users. The search head for admins once or twice a week will need to have splunk service restarted in order to read from the indexer again. Looking in the distributed search peers settings it shows the indexer as "Up", "Healthy" and with no failed health checks. When searching from this indexer on any index, no results are returned, almost as if it cannot talk to the indexer. Running a tcpdump on both the indexer and problematic search head shows bidirectional traffic between the two hosts with no issues.

Our other search head is still working fine and returning results so we know data is still being indexed properly. Is there anything specific I should be looking at to track down this issue?

0 Karma

gstefancyk
Path Finder

Thanks Giuseppe, I don't see any issue with indexing from DMC. As I noted our other search head is still able to search with no issues. Our indexer storage should not be a problem as we are running Raid10 on 8 1.6TB SSD's. The indexer and search heads are physical machines.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi gstefancyk,
check (eventually using Distributed Monitoring Console) if on your Indexers there were problems on indexing queue.
I had the same problem caused by a very slow storage.

In addition I found this problem on a server that used vm-ware esxi 6 instead I didn't have problems with indexers that used vm-ware esxi 5.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

sorry.
open a case to Splunk support.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...