Splunk Search

DISPATCHCOMM:PEER_PIPE_EXCEPTION__%s message=Search results might be incomplete: the search process on the peer

HarishSamudrala
Loves-to-Learn

Hello Splunk team...
I am facing this issue while we run any searches on my splunk setup., can you help me on how we can fix this..


02-29-2024 06:58:53.370 ERROR DispatchThread [4125 phase_1] - code=10 error=""
02-29-2024 06:58:53.370 ERROR ResultsCollationProcessor [4125 phase_1] - SearchMessage orig_component=ResultsCollationProcessor sid=1709189933.399443_**** message_key=DISPATCHCOMM:PEER_PIPE_EXCEPTION__%s message=Search results might be incomplete: the search process on the peer:  ended prematurely. Check the peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.

 

Thank you..

Labels (1)
0 Karma

HarishSamudrala
Loves-to-Learn

@kiran_panchavat...
Sorry for the late reply...we tried checking all the steps..peerlogs, license., oom issue...could not find anything wrong and all were good.

So we tried a rolling restrat of the SH cluster ...that fixed the issue and errors were gone 🙂
Thank you for your time on helping with this..

0 Karma

kiran_panchavat
Motivator

@HarishSamudrala 

The error message you provided indicates that search results might be incomplete due to the search process ending prematurely on the peer. 

Check Peer Logs:

Look into the peer log files, specifically:

$SPLUNK_HOME/var/log/splunk/splunkd.log

The search.log for the particular search.

Examine these logs for any relevant error messages or clues about what caused the premature termination.

Memory and Resource Constraints:

Ensure that the peer has sufficient resources (CPU, memory, disk space) to handle the search workload.

Sometimes, insufficient resources can lead to premature search process termination.

Consider monitoring system resource usage during search execution.

License Considerations:

If you’re using a trial Splunk Enterprise distributed deployment, each instance must use its own self-generated Enterprise Trial license.

In contrast, a distributed deployment running a Splunk Enterprise license requires configuring a license master to host all licenses.

Check for OOM Killer Events:

Review /var/log/messages on the peer for any Out-of-Memory (OOM) Killer events.

Insufficient memory can cause processes to terminate unexpectedly.

Increase ulimits for Open Files:

If you haven’t already, consider increasing the ulimits for open files on the indexers.

For example, set the ulimit to the recommended 64000 (initially it might be set to 4096).

Review Configuration:

Verify that the configuration of your search head, indexers, and forwarders is correct.

Ensure that the search head can communicate with the peer properly.

Remember to investigate the specific details in the logs to pinpoint the root cause. If you encounter any specific error messages or need further assistance, feel free to share additional details. 

Solved: Search results might be incomplete: the search pro... - Splunk Community 

https://community.splunk.com/t5/Splunk-Search/Search-results-might-be-incomplete-the-search-process-... 

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.
0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...