Getting Data In

Is there a way to change collection interval for HTTP Event Collector?

splunkguy0342
New Member

I am using HTTP Event Collector to collect Symantec ATP logs, my current ingest rate varies based on log size. It is typically around 2000-5000 logs at a rate of every 1 minute. My log source is generating between 1.5 M -3 M events per day. The collector is averaging about 480k-960k events per day. This is putting me into a logging deficit where I am unable to keep up with log generation. I am looking to change the interval to every 5 seconds or vastly increase the collection rate. I am for the most part default settings, the event collector is running on a heavy forwarder and forwarding to an indexer cluster, we have tried pointing to a single indexer but performance did not change.

0 Karma

lguinn2
Legend

Where is the actual bottleneck on the heavy forwarder: network, memory, CPU?

Forwarding to an indexer cluster should not be slower than forwarding to a single indexer, so I am not surprised that didn't help.
There is no "collection interval" on the heavy forwarder; it should be able to "collect" the events asynchronously as they are sent over http/https.

My guess is that you may be exceeding the bandwidth of a single event collector. Have you considered using 2 heavy forwarders and having the sender switch between them?

If the resources on the heavy forwarder are not being taxed, then perhaps the sender trying to exceed its output bandwidth.

0 Karma
Get Updates on the Splunk Community!

Automatic Discovery Part 1: What is Automatic Discovery in Splunk Observability Cloud ...

If you’ve ever deployed a new database cluster, spun up a caching layer, or added a load balancer, you know it ...

Real-Time Fraud Detection: How Splunk Dashboards Protect Financial Institutions

Financial fraud isn't slowing down. If anything, it's getting more sophisticated. Account takeovers, credit ...

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...