Hello, I had random problems when i try execute a search, following job inspector, search.log show error like that: ERROR SearchResultTransaction - Got status 502 from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47
ERROR SearchResultTransaction - Got status 502 from https://y.y.y.y:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47 ERROR SearchResultParser - HTTP error status message from https://x.x.x.x:18089/services/streams/search?sh_sid=scheduler__nobody_SVNPX2F1ZGl0__RMD5e21e138c00642ff9_at_1609913400_774_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47: Error connecting: Connect Timeout SearchResultCollator - Failure received on retry collector. _unresolvedRetries=1 ERROR HttpListener - Exception while processing request from SH_IP:46819 for /services/search/jobs/searchhead_1609914001.2117_BF9B9E7A-91AD-4585-AFE9-DA7D03E57F47/search.log: Connection closed by peer I've confirmed that connection between search head and search peer is still normal but sometimes the search peer show error "Broken pipe or Connection closed by peer" and on search head show "502 Status" when connect to search peer. I used Splunk 7.3.7.1 with OS CentOS 7 and some tunning configuration: net.core.somaxconn = 1024 net.ipv4.tcp_max_syn_backlog = 4096 server.conf on search pear [httpServer] maxSockets = 400000 maxThreads = 150000 keepAliveIdleTimeout = 7200 busyKeepAliveIdleTimeout = 12 limits.conf on search head [search] max_chunk_queue_size = 30000000 Anybody can help me to fix this or understand why is happening this. Thanks.
... View more