When running a search the following error will appear in the job inspector. Users get this message intermittently on searches. No results can be returned.
10-18-2022 11:00:22.349 ERROR DispatchThread [3247729 phase_1] - code=10 error="" 10-18-2022 11:00:22.349 ERROR ResultsCollationProcessor [3247729 phase_1] - SearchMessage orig_component= sid=1666090813.341131_7E89B3C6-34D5-44DA-B19C-E6A755245D39 message_key=DISPATCHCOMM:PEER_PIPE_EXCEPTION__%s message=Search results might be incomplete: the search process on the peer:pldc1splindex1 ended prematurely. Check the peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.
The message.conf shows
message = Search results might be incomplete: the search process on the local peer:%s ended prematurely.
action = Check the local peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.
severity = warn
I also have Splunk Alerts that are showing false positives, the alert search is retuning no results but the Splunk sourcetype=scheduler is sending out emails with success?
Is this related?
What does this mean? PEER_PIPE_EXCEPTION__S
Splunk Enterprise OnPrem version 9.0.1 on a distributed environment.
Did you look at splunkd.conf on the peer as well as search.log like the error suggested? What did you find there?
Messages.conf is not a troubleshooting aid. It's for assigning severities to log messages. "PEER_PIPE_EXCEPTION__S" identifies the type of error encountered.
I found this in the splunkd.log on one of the splunk indexers at the time of the error message
10-18-2022 11:00:17.141 +0000 ERROR SearchProcessRunner [2379030 PreforkedSearchesManager-0] - preforked process=0/437059 hung up
10-18-2022 11:00:17.163 +0000 WARN SearchProcessRunner [2379030 PreforkedSearchesManager-0] - preforked process=0/437059 status=killed, signum=9, signame="Killed", coredump=0, utime_sec=1.672967, stime_sec=0.285628, max_rss_kb=207912, vm_minor=72863, fs_r_count=6352, fs_w_count=456, sched_vol=407, sched_invol=1431
Is this a Swap memory issue?
Indeed there was such messages in DMESG on the Indexers.
I've had to Disable Swap Memory : sqapoff -a
and done a rolling restart across the indexers.
Thanks for your time on this.