How to use HTTP Event Collector(HEC) to setup an app to log to Splunk?

[mhelmers]

I am trying to set up an app to log to Splunk but I have a few (basic) questions.
First I was just going to write the events to a log on disk and have Splunk continuously monitor the file but I would like to steer away from writing to disk if possible because we store personal information. I like the HTTP Event Collector option but I am concerned with what happens if there are network connectivity issues or some other downtime. I see that there is some resilience built into the appenders but does that mean that the events will be stored in some queue until a connection can be re-established? How big is that queue if there is one, and do events get dropped if the queue fills up? If I cant drop a single event log is this option even viable, or would it still be recommended to write to disk?

Also talking with one of our sys admins he mentioned that he would have to set up a dedicated HEC forwarder, would this be the case? I thought that using the HEC option negated having to use a forwarder, or am I completely misunderstanding what needs to happen here to implement this?

[starcher]

You run HEC on a heavy forwarder. If you need to scale you run multiples behind a load balancer. How much the HFs hold in pipelines depends on how the admin configured the heavy forwarders.

However most people just scale enough HEC capacity to handle the load. Doing things acknowledgements are hard to implement for the application developer making the logs.

http://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Data/UsetheHTTPEventCollector
http://dev.splunk.com/view/event-collector/SP-CAAAE7G

https://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Protectagainstlossofin-flightdata