topic Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"? in Splunk Enterprise

What is this error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

jencot01 — Fri, 02 Dec 2022 00:34:26 GMT

I have recently run into an issue with multiple "WARN HttpListener [HttpDedicatedIoThread:0] Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe" for each Indexer in the Index cluster.

There is a SH cluster, and a standalone SH. The standalone houses an app that does heavy backend searching. And the majority of the errors are from the standalone. When looking at the "I/O Operations per second" and "Storage I/O Saturation (cold/hot/opt)" from the Monitoring Console all instances are below 1%.

I am not sure which settings to adjust to fix this error. Would adjusting the maxSockets and/or maxThreads in the server.conf help? Currently they are both set to default. Or should I be looking at values in limits.conf?

This is happening on version 9.0.1. Any suggestions to help solve this would be much appreciated. Thanks!

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

jencot01 — Thu, 20 Oct 2022 15:31:55 GMT

To add to the above, I have also found this error in splunkd.log on the indexers:

ERROR SearchProcessRunner PreforkedSearchesManager-0 - preforked process = 0/253717 hung up

This error aligns with the "broken pipe" error on my post above. There are no skipped searches.

I'm at a loss to what I steps I should take next to troubleshoot this issue. Again, any help/suggestions would be appreciated.

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

anirban_td — Thu, 01 Dec 2022 23:50:14 GMT

We recently upgraded our deployment (IDX Cluster + SH Cluster) from 8.2.6 to 9.0.1

Since the upgrade, we see a huge number of warnings for :

WARN HttpListener [8466 HttpDedicatedIoThread-7] - Socket error from <ip:port> while accessing <URI>: Broken pipe

Along with this, there are huge number of errors for Captain Disconnected, "This member has marked the connection to the search head captain as down", timeouts, and skipped searches.

These errors/warnings typically occur during peak loads.

As such we found that THP was enabled on few of SH Cluster members. ulimits were set to Splunk recommended values, but we are in the process of increasing them in our prod env.

Further troubleshooting pending, Will get back with updates.

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

tfellinger — Sun, 01 Jan 2023 08:08:42 GMT

Did any of you ever fixed this issue? If so, how?

Thanks!

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

jlc00 — Mon, 02 Jan 2023 18:35:35 GMT

I was never able to fix the issue. Still trying to figure it out.

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

mika703 — Fri, 21 Jul 2023 06:08:10 GMT

Same happened to our Infrastructure after Upgrading from 8.2.9 -> 9.0.5. Has anybody figured out whats causing this messages and what the impact is?

07-21-2023 07:42:16.414 +0200 ERROR SearchProcessRunner [24412 PreforkedSearchesManager-0] - preforked process=063487 hung up

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

Gregster66 — Wed, 18 Oct 2023 13:49:59 GMT

Seeing the same after an upgrade from v8 to v9.0.6.
I'm suspecting something went wrong during the upgrade but don't have any solid evidence yet.
Did anyone manage to get to the bottom of this ?

Re: Why the error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

mika703 — Wed, 18 Oct 2023 15:20:50 GMT

We are still having these ERROR Messages since the upgrade. Never found some evidence or root cause 😕