Getting Data In

Is there a way to change collection interval for HTTP Event Collector?

splunkguy0342
New Member

I am using HTTP Event Collector to collect Symantec ATP logs, my current ingest rate varies based on log size. It is typically around 2000-5000 logs at a rate of every 1 minute. My log source is generating between 1.5 M -3 M events per day. The collector is averaging about 480k-960k events per day. This is putting me into a logging deficit where I am unable to keep up with log generation. I am looking to change the interval to every 5 seconds or vastly increase the collection rate. I am for the most part default settings, the event collector is running on a heavy forwarder and forwarding to an indexer cluster, we have tried pointing to a single indexer but performance did not change.

0 Karma

lguinn2
Legend

Where is the actual bottleneck on the heavy forwarder: network, memory, CPU?

Forwarding to an indexer cluster should not be slower than forwarding to a single indexer, so I am not surprised that didn't help.
There is no "collection interval" on the heavy forwarder; it should be able to "collect" the events asynchronously as they are sent over http/https.

My guess is that you may be exceeding the bandwidth of a single event collector. Have you considered using 2 heavy forwarders and having the sender switch between them?

If the resources on the heavy forwarder are not being taxed, then perhaps the sender trying to exceed its output bandwidth.

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...