topic How to fix error: Server is busy - HTTP Event Collector in Monitoring Splunk

How to fix error: Server is busy - HTTP Event Collector

robertlynch2020 — Tue, 08 Mar 2022 05:02:39 GMT

We are sending in Opentelemtory metrics into Splunk via HTTP Event Collector.

However, we got the following errors the other days "server is busy" . I can see the data did come in at that time, but it gets retried so that explains that.

How do I stop this from happening in the future?

Another question is what is the max throughput Splunk can take in via HTTP?

The below code came from the OP - Python scripts

2022-03-04T19:41:36.125+0100 info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "splunk_hec/logs", "error": "Post \https://dell425srv:9088/services/collector\: context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "interval": "5.6081835s"}

Thanks in advance

Rob

Re: Server is busy - HTTP Event Collector - how do i fix it

somesoni2 — Mon, 07 Mar 2022 18:47:00 GMT

I would tune Splunk HEC so that it's thruput is optimized. A good read is here: https://conf.splunk.com/files/2017/slides/measuring-hec-performance-for-fun-and-profit.pdf

Re: Server is busy - HTTP Event Collector - how do i fix it

robertlynch2020 — Tue, 08 Mar 2022 12:45:31 GMT

This is good information, and thanks

I will look at it and hopefully it will help me.

Another question to ask is, do you think increased HEC traffic on an non-optmised Splunk can cause Splunk to crash with "inotify cannot be used, reverting to polling: Too many open files". I am starting to get that now and the only major change is the new traffic from HEC.

Thanks

Rob