Splunk Enterprise

What is this error "Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe"?

jencot01
Loves-to-Learn Lots

I have recently run into an issue with multiple "WARN HttpListener [HttpDedicatedIoThread:0] Socket error from [Search Head IP] while accessing /services/streams/search: broken pipe" for each Indexer in the Index cluster.

There is a SH cluster, and a standalone SH. The standalone houses an app that does heavy backend searching. And the majority of the errors are from the standalone. When looking at the "I/O Operations per second" and "Storage I/O Saturation (cold/hot/opt)" from the Monitoring Console all instances are below 1%.

I am not sure which settings to adjust to fix this error. Would adjusting the maxSockets and/or maxThreads in the server.conf help? Currently they are both set to default. Or should I be looking at values in limits.conf?

This is happening on version 9.0.1. Any suggestions to help solve this would be much appreciated. Thanks!

Tags (2)
0 Karma

anirban_td
Explorer

We recently upgraded our deployment (IDX Cluster + SH Cluster) from 8.2.6 to 9.0.1

Since the upgrade, we see a huge number of warnings for :

WARN HttpListener [8466 HttpDedicatedIoThread-7] - Socket error from <ip:port> while accessing <URI>: Broken pipe

Along with this, there are huge number of errors for Captain Disconnected, "This member has marked the connection to the search head captain as down", timeouts, and skipped searches.

These errors/warnings typically occur during peak loads.

As such we found that THP was enabled on few of SH Cluster members. ulimits were set to Splunk recommended values, but we are in the process of increasing them in our prod env.

Further troubleshooting pending, Will get back with updates.

0 Karma

jencot01
Loves-to-Learn Lots

To add to the above, I have also found this error in splunkd.log on the indexers:

ERROR SearchProcessRunner  PreforkedSearchesManager-0   - preforked process = 0/253717  hung up

This error aligns with the "broken pipe" error on my post above.  There are no skipped searches.

I'm at a loss to what I steps I should take next to troubleshoot this issue.   Again, any help/suggestions would be appreciated.

0 Karma

mika703
Engager

Same happened to our Infrastructure after Upgrading from 8.2.9 -> 9.0.5. Has anybody figured out whats causing this messages and what the impact is?

 

07-21-2023 07:42:16.414 +0200 ERROR SearchProcessRunner [24412 PreforkedSearchesManager-0] - preforked process=063487 hung up

0 Karma

Gregster66
Engager

Seeing the same after an upgrade from v8 to v9.0.6.
I'm suspecting something went wrong during the upgrade but don't have any solid evidence yet. 
Did anyone manage to get to the bottom of this ?

0 Karma

tfellinger
New Member

Did any of you ever fixed this issue? If so, how?

Thanks!

0 Karma

jlc00
Loves-to-Learn Lots

I was never able to fix the issue.  Still trying to figure it out.

0 Karma

mika703
Engager

We are still having these ERROR Messages since the upgrade. Never found some evidence or root cause 😕 

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...