https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339558#M8641 <P>How does the system resource usage on your search heads look? And is there a large number of concurrent searches? You may want to take a look at the "Search Activity: Instance" dashboard in the Monitoring Console to see if there are searches running using a large amount of memory.</P> Tue, 30 Jan 2018 20:45:36 GMT traxxasbreaker 2018-01-30T20:45:36Z https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339557#M8640 <P>Hello,</P> <P>I recently joined a project at a place with an ES Search head and a few ad-hoc ones. Apparently, there has been a new issue that started a few days before I got here. The ES SH (others are fine) will go unresponsive for web browsing until it is restarted. The web daemon and splunkd are still running. Alerts and searches are still running.</P> <P>Splunkd is just <EM>bombarded</EM> with <CODE>"ERROR HttpClientRequest - HTTP client error=Read Timeout while connecting to server"</CODE> messages and pretty much nothing else. I'm trying to help them figure out what the issue is, but I've never seen something like this happen.</P> <P>Any thoughts?</P> Tue, 30 Jan 2018 16:57:38 GMT https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339557#M8640 jadamsplunk 2018-01-30T16:57:38Z https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339558#M8641 <P>How does the system resource usage on your search heads look? And is there a large number of concurrent searches? You may want to take a look at the "Search Activity: Instance" dashboard in the Monitoring Console to see if there are searches running using a large amount of memory.</P> Tue, 30 Jan 2018 20:45:36 GMT https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339558#M8641 traxxasbreaker 2018-01-30T20:45:36Z https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339559#M8642 <P>That was my first thought as well, but I can't access anything on the SH when it's like that. Looking at the charts historically for when it happens, though, I don't see any spikes.</P> Tue, 30 Jan 2018 20:47:33 GMT https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339559#M8642 jadamsplunk 2018-01-30T20:47:33Z https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339560#M8643 <P>if the ES search head forwards its data to indexer layer, you supposed to be able to query its status through another search head: index = _internal host=ES_Searhc_Head</P> Tue, 29 Sep 2020 17:52:37 GMT https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339560#M8643 adonio 2020-09-29T17:52:37Z https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339561#M8644 <P>Hi did you find out what caused this error?</P> Mon, 20 Aug 2018 16:53:34 GMT https://community.splunk.com/t5/Security/What-could-cause-splunkd-log-to-be-flooded-with-quot-ERROR/m-p/339561#M8644 mrtolu6 2018-08-20T16:53:34Z