Getting Data In

How to use HTTP Event Collector(HEC) to setup an app to log to Splunk?

mhelmers
New Member

I am trying to set up an app to log to Splunk but I have a few (basic) questions.
First I was just going to write the events to a log on disk and have Splunk continuously monitor the file but I would like to steer away from writing to disk if possible because we store personal information. I like the HTTP Event Collector option but I am concerned with what happens if there are network connectivity issues or some other downtime. I see that there is some resilience built into the appenders but does that mean that the events will be stored in some queue until a connection can be re-established? How big is that queue if there is one, and do events get dropped if the queue fills up? If I cant drop a single event log is this option even viable, or would it still be recommended to write to disk?

Also talking with one of our sys admins he mentioned that he would have to set up a dedicated HEC forwarder, would this be the case? I thought that using the HEC option negated having to use a forwarder, or am I completely misunderstanding what needs to happen here to implement this?

0 Karma

starcher
SplunkTrust
SplunkTrust

You run HEC on a heavy forwarder. If you need to scale you run multiples behind a load balancer. How much the HFs hold in pipelines depends on how the admin configured the heavy forwarders.

However most people just scale enough HEC capacity to handle the load. Doing things acknowledgements are hard to implement for the application developer making the logs.

http://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Data/UsetheHTTPEventCollector
http://dev.splunk.com/view/event-collector/SP-CAAAE7G

https://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Protectagainstlossofin-flightdata

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...