Indexer got 502 error

huuphuc1807 — Wed, 06 Jan 2021 06:43:45 GMT

Hello,

I had random problems when i try execute a search, following job inspector, search.log show error like that:

ERROR SearchResultTransaction - Got status 502 from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47
ERROR SearchResultTransaction - Got status 502 from https://y.y.y.y:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47

ERROR SearchResultParser - HTTP error status message from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47: Error connecting: Connect Timeout

SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1
ERROR HttpListener - Exception while processing request from SH_IP:46819 for /services/search/jobs/searchhead_1609914001.2117_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47/search.log: Connection closed by peer

I've confirmed that connection between search head and search peer is still normal but sometimes the search peer show error "Broken pipe or Connection closed by peer" and on search head show "502 Status" when connect to search peer.
I used Splunk 7.3.7.1 with OS CentOS 7 and some tunning configuration:

net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 4096

server.conf on search pear
[httpServer]
maxSockets = 400000
maxThreads = 150000
keepAliveIdleTimeout = 7200
busyKeepAliveIdleTimeout = 12

limits.conf on search head
[search]
max_chunk_queue_size = 30000000

Anybody can help me to fix this or understand why is happening this.

Thanks.

Re: Indexer got 502 error

sanderdenheijer — Tue, 26 Jan 2021 15:55:29 GMT

Hi huuphuc1807,

Did you find the root cause/fix by any chance? We are experiencing the same issue using Splunk 7.3.4 on RHEL 7 servers.

Thanks,

Sander

Re: Indexer got 502 error

huuphuc1807 — Mon, 01 Feb 2021 07:54:32 GMT

Hi,

Not yet, i still got that error. Have you had idea or solution?

Regards,

topic Re: Indexer got 502 error in Splunk Enterprise

Indexer got 502 error

Re: Indexer got 502 error

Re: Indexer got 502 error