Getting Data In

How to use HTTP Event Collector(HEC) to setup an app to log to Splunk?

mhelmers
New Member

I am trying to set up an app to log to Splunk but I have a few (basic) questions.
First I was just going to write the events to a log on disk and have Splunk continuously monitor the file but I would like to steer away from writing to disk if possible because we store personal information. I like the HTTP Event Collector option but I am concerned with what happens if there are network connectivity issues or some other downtime. I see that there is some resilience built into the appenders but does that mean that the events will be stored in some queue until a connection can be re-established? How big is that queue if there is one, and do events get dropped if the queue fills up? If I cant drop a single event log is this option even viable, or would it still be recommended to write to disk?

Also talking with one of our sys admins he mentioned that he would have to set up a dedicated HEC forwarder, would this be the case? I thought that using the HEC option negated having to use a forwarder, or am I completely misunderstanding what needs to happen here to implement this?

0 Karma

starcher
Influencer

You run HEC on a heavy forwarder. If you need to scale you run multiples behind a load balancer. How much the HFs hold in pipelines depends on how the admin configured the heavy forwarders.

However most people just scale enough HEC capacity to handle the load. Doing things acknowledgements are hard to implement for the application developer making the logs.

http://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Data/UsetheHTTPEventCollector
http://dev.splunk.com/view/event-collector/SP-CAAAE7G

https://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Protectagainstlossofin-flightdata

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...