Indexer got 502 error

[huuphuc1807]

Hello,

I had random problems when i  try execute a search, following job inspector, search.log show error like that:

ERROR SearchResultTransaction - Got status 502 from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47
ERROR SearchResultTransaction - Got status 502 from https://y.y.y.y:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c006...

ERROR SearchResultParser - HTTP error status message from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47: Error connecting: Connect Timeout

SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1
ERROR HttpListener - Exception while processing request from SH_IP:46819 for /services/search/jobs/searchhead_1609914001.2117_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47/search.log: Connection closed by peer

I've confirmed that connection between search head and search peer is still normal but sometimes the search peer show error "Broken pipe or Connection closed by peer" and on search head show "502 Status" when connect to search peer.
I used Splunk 7.3.7.1 with OS CentOS 7 and some tunning configuration:

net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 4096

server.conf on search pear
[httpServer]
maxSockets = 400000
maxThreads = 150000
keepAliveIdleTimeout = 7200
busyKeepAliveIdleTimeout = 12

limits.conf on search head
[search]
max_chunk_queue_size = 30000000

Anybody can help me to fix this or understand why is happening this.

Thanks.

[sanderdenheijer]

Hi huuphuc1807,

Did you find the root cause/fix by any chance? We are experiencing the same issue using Splunk 7.3.4 on RHEL 7 servers.

Thanks,

Sander

[huuphuc1807]

Hi,

Not yet, i still got that error. Have you had idea or solution?

Regards,