Hello,
I had random problems when i try execute a search, following job inspector, search.log show error like that:
ERROR SearchResultTransaction - Got status 502 from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47
ERROR SearchResultTransaction - Got status 502 from https://y.y.y.y:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c006...
ERROR SearchResultParser - HTTP error status message from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47: Error connecting: Connect Timeout
SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1
ERROR HttpListener - Exception while processing request from SH_IP:46819 for /services/search/jobs/searchhead_1609914001.2117_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47/search.log: Connection closed by peer
I've confirmed that connection between search head and search peer is still normal but sometimes the search peer show error "Broken pipe or Connection closed by peer" and on search head show "502 Status" when connect to search peer.
I used Splunk 7.3.7.1 with OS CentOS 7 and some tunning configuration:
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 4096
server.conf on search pear
[httpServer]
maxSockets = 400000
maxThreads = 150000
keepAliveIdleTimeout = 7200
busyKeepAliveIdleTimeout = 12
limits.conf on search head
[search]
max_chunk_queue_size = 30000000
Anybody can help me to fix this or understand why is happening this.
Thanks.
