Splunk Enterprise

Indexer got 502 error

huuphuc1807
New Member

Hello,

I had random problems when i  try execute a search, following job inspector, search.log show error like that:

ERROR SearchResultTransaction - Got status 502 from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47
ERROR SearchResultTransaction - Got status 502 from https://y.y.y.y:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c006...
ERROR SearchResultParser - HTTP error status message from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47: Error connecting: Connect Timeout
SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1
ERROR HttpListener - Exception while processing request from SH_IP:46819 for /services/search/jobs/searchhead_1609914001.2117_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47/search.log: Connection closed by peer

I've confirmed that connection between search head and search peer is still normal but sometimes the search peer show error "Broken pipe or Connection closed by peer" and on search head show "502 Status" when connect to search peer.
I used Splunk 7.3.7.1 with OS CentOS 7 and some tunning configuration:

net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 4096

server.conf on search pear
[httpServer]
maxSockets = 400000
maxThreads = 150000
keepAliveIdleTimeout = 7200
busyKeepAliveIdleTimeout = 12

limits.conf on search head
[search]
max_chunk_queue_size = 30000000

Anybody can help me to fix this or understand why is happening this.

Thanks.

Labels (1)
0 Karma

sanderdenheijer
Explorer

Hi huuphuc1807,

Did you find the root cause/fix by any chance? We are experiencing the same issue using Splunk 7.3.4 on RHEL 7 servers.

Thanks,

Sander

0 Karma

huuphuc1807
New Member

Hi,

Not yet, i still got that error. Have you had idea or solution?

Regards,

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...