topic Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe in Monitoring Splunk

HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Sat, 06 Jun 2020 02:05:54 GMT

I am frequently getting warning for Socket.

WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody//data/inputs/rest/*/: Broken pipe
source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

i have checked below answers, but unable to resolve my query:
https://answers.splunk.com/answers/105292/what-is-the-cause-of-these-socket-errors-reported-in-splunkd-log-since-upgrading-to-6-0.html

limits:
maxthread=0
maxsocket=0

INFO loader - Limiting REST HTTP server to 1365 threads
INFO loader - Limiting REST HTTP server to 1365 sockets

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 47488
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 8192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

/etc/security/limits.conf
* hard nofile 64000
* hard nproc 8192
* hard fsize -1

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Wed, 29 May 2019 07:03:09 GMT

Hi @sarvesh_11,

You're open files (-n)limit is still at 1024. You should increase that for the Splunk user, and should also increase the soft limit.

Try something like this in your limits.conf file :

*    hard    nofile     64000
*    soft    nofile     64000
*    hard    nproc     8192
*    soft    nproc     8192
*    hard    fsize      -1  
*    soft    fsize      -1

Or like this if that doesn't work :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Troubleshooting/ulimitErrors#Set_limits_using_the_.2Fetc.2Fsystemd_configuration_files

Cheers,
David

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 29 May 2019 07:19:54 GMT

Hey @DavidHourani ,
Thanks for dropping by.
Later posting the question i soon realized that, and made the changes in limits.conf

@splunk hard nofile 64000
@splunk hard nproc 8192
@splunk hard fsize -1
@splunk soft nofile 64000
@splunk soft nproc 8192
@splunk soft fsize -1

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 47488
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 64000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 8192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

still the issue persist.
Also, just while checking the app owner, i can see different users(root,splunk,), so shall i change the owner to splunk for all the apps?

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Wed, 29 May 2019 07:23:02 GMT

you can use * for now for the user in /etc/security/limits.conf. And restart splunk after changing that setting and check the _internal logs for ulimit to double check that Splunk also reads the right ulimits. if you run the following search you should get the relevant lines :

index=_internal  component=ulimit

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 29 May 2019 07:43:54 GMT

@DavidHourani
Thanks much man!
The changes are replicated, i shall get back to you if still the "Break pipe" error is there

Thanks alot for your prompt response 🙂

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 30 Sep 2020 00:43:38 GMT

@DavidHourani ,
Hey David, the error is still coming. THe changes of lmits are reflected, but still

Socket error from 127.0.0.1 while accessing /servicesNS/nobody//data/inputs/rest/http_connections_*/: Broken pipe.

Or i need to change maxSockets and maxThreads to negative integer?
currently it is 0
Any more remedies you have for this?

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Wed, 29 May 2019 08:16:43 GMT

yeah try the maxSockets and maxThreads to -1 see if it helps, any other errors you're getting ?

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 30 Sep 2020 00:43:40 GMT

Yeah, few more and related to that only:

"ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 500 Server Error: Internal Server Error"

"ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/Splunk_TA_box/bin/box_service.py" InsecureRequestWarning)"

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Wed, 29 May 2019 08:37:45 GMT

okay try out : maxSockets and maxThreads and lets see

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 29 May 2019 08:46:42 GMT

@DavidHourani
Well for doing this and restarting services again, i have to take maintenance window from business.
Ideally that should not happen right? Setting to -1 will impact server's performance also.

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Wed, 29 May 2019 08:49:33 GMT

yeah.... ideally you would need a reasonable limit to avoid breaking the server...are you using a single SH ?

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Wed, 29 May 2019 09:11:36 GMT

yaa, standalone Search Head

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Mon, 03 Jun 2019 08:25:16 GMT

Hey @DavidHourani ,
Modifying maxSockets and maxThread to -1, has not suffice my issue. I can still see the Broken Pipe messages.

Also, while browsing for such issue , found a link where people are struggling since long from such errors, strangely none of the Splunk Employee tried to resolve it.
FYR.. https://answers.splunk.com/answers/105292/what-is-the-cause-of-these-socket-errors-reported-in-splunkd-log-since-upgrading-to-6-0.html

Seems this is the loop hole in splunk older version, as SPL-82389 known issue has not been closed and addressed since 6.2.9 version.

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

DavidHourani — Mon, 03 Jun 2019 15:02:16 GMT

Which Splunk version are you running ? Have you tried reaching out for Splunk support, you MIGHT be on a bug then...

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Tue, 04 Jun 2019 02:53:43 GMT

We are currently on 6.6.3
Not yet, we are in plan to upgrade in near time.

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

dijikul — Fri, 06 Mar 2020 02:50:45 GMT

Why was the above answer marked as accepted when on June 03 2019 @sarvesh_11 states the issue persists?

We're seeing this too. Was this issue resolved?

Re: HttpListener - Socket error from 127.0.0.1 while accessing : Broken Pipe

sarvesh_11 — Fri, 06 Mar 2020 08:38:29 GMT

Hey @dijikul
We have upgraded splunk enterprise version and were able to solve this issue.
We upgraded it to 7.3