Getting Data In

Is there a way to change collection interval for HTTP Event Collector?

splunkguy0342
New Member

I am using HTTP Event Collector to collect Symantec ATP logs, my current ingest rate varies based on log size. It is typically around 2000-5000 logs at a rate of every 1 minute. My log source is generating between 1.5 M -3 M events per day. The collector is averaging about 480k-960k events per day. This is putting me into a logging deficit where I am unable to keep up with log generation. I am looking to change the interval to every 5 seconds or vastly increase the collection rate. I am for the most part default settings, the event collector is running on a heavy forwarder and forwarding to an indexer cluster, we have tried pointing to a single indexer but performance did not change.

0 Karma

lguinn2
Legend

Where is the actual bottleneck on the heavy forwarder: network, memory, CPU?

Forwarding to an indexer cluster should not be slower than forwarding to a single indexer, so I am not surprised that didn't help.
There is no "collection interval" on the heavy forwarder; it should be able to "collect" the events asynchronously as they are sent over http/https.

My guess is that you may be exceeding the bandwidth of a single event collector. Have you considered using 2 heavy forwarders and having the sender switch between them?

If the resources on the heavy forwarder are not being taxed, then perhaps the sender trying to exceed its output bandwidth.

0 Karma
Get Updates on the Splunk Community!

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

New Release | Splunk Cloud Platform 10.1.2507

Hello Splunk Community!We are thrilled to announce the General Availability of Splunk Cloud Platform 10.1.2507 ...

🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2

🗣 You Spoke, We Listened  Audit Trail v2 wasn’t written in isolation—it was shaped by your voices.  In ...