Getting Data In

Is there a way to change collection interval for HTTP Event Collector?

splunkguy0342
New Member

I am using HTTP Event Collector to collect Symantec ATP logs, my current ingest rate varies based on log size. It is typically around 2000-5000 logs at a rate of every 1 minute. My log source is generating between 1.5 M -3 M events per day. The collector is averaging about 480k-960k events per day. This is putting me into a logging deficit where I am unable to keep up with log generation. I am looking to change the interval to every 5 seconds or vastly increase the collection rate. I am for the most part default settings, the event collector is running on a heavy forwarder and forwarding to an indexer cluster, we have tried pointing to a single indexer but performance did not change.

0 Karma

lguinn2
Legend

Where is the actual bottleneck on the heavy forwarder: network, memory, CPU?

Forwarding to an indexer cluster should not be slower than forwarding to a single indexer, so I am not surprised that didn't help.
There is no "collection interval" on the heavy forwarder; it should be able to "collect" the events asynchronously as they are sent over http/https.

My guess is that you may be exceeding the bandwidth of a single event collector. Have you considered using 2 heavy forwarders and having the sender switch between them?

If the resources on the heavy forwarder are not being taxed, then perhaps the sender trying to exceed its output bandwidth.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...