HEC is at the end of the day, a web server. For the most part, it only acts as a simple listener of HTTP(S) POST messages, and as such, it does not guarantee that a sent message was received. The sender is expected to use HTTP protocol conventions such as the status code to determine if a message was received, and if not, do whatever it is that the sender wants to do. All of this is by design, and it works exactly as a web server is expected to work. One exception to the above is the HEC Indexer Acknowledgement feature, which requires the developer to implement client-side code. More details are here: https://docs.splunk.com/Documentation/Splunk/7.3.2/Data/AboutHECIDXAck If “guaranteed messaging” is a requirement, I suggest looking at one of these options: (In AWS) Kinesis Data Firehose which has a reliable pipeline to Splunk Enterprise or Splunk Cloud (behind the scenes, the nozzle uses HEC with the indexer acknowledgement feature enabled) (Not AWS) Use a any queueing system which provides a guaranteed delivery feature. Then you would need to use another process to dequeue those messages and send to Splunk. Use a logging library (e.g. log4j) which supports multiple destinations, and add a disk buffer as a fallback. To address momentary connection issues, Splunk has added logic to a handful of client libraries and other tools that will handle things like: batching, buffering, and retrying. Those resources are published at http://dev.splunk.com/view/SP-CAAAFD2, and are available as open source. Realize that these clients will typically use a small (configurable) memory buffer which can quickly be filled (depending on the size of the buffer and the length of the outage). The available logging libraries are: Java Javascript .NET We also provide full open source SDKs which include logic for many tasks outside of logging, as well as code samples for these languages: C# Java Javascript Python Lastly, the Docker Logging Driver does include batch, buffer, and retry logic. By default, it will buffer 10,000 lines, as documented here: https://github.com/splunk/docker-logging-plugin#advanced-options---environment-variables ... View more

Answering my own question after research and consulting with Microsoft: When an app writes to standard out, use the Docker Logging Driver for Splunk, which sends data to HEC. However, this is not as commonly done on Windows. When using logging libraries such as Log4Net or Log4J, or Splunk's logging tools, these can be easily reconfigured to send data to HEC. Event logs must be either pushed or pulled from the container, to another system. One approach that makes sense in a containerized environment is to use Windows Event Log Forwarding (WEF) to push logs from containers to the host OS. Use the Universal Forwarder on the host OS to collect these logs. Perfmon (performance metrics) objects may exist on the host which can be collected via UF. (This item needs further research.) Log files can be mounted using shared volumes (a WSC/docker feature, example here) to a separate container, or the host OS, and then use the standard file monitor feature in Splunk. More specific WSC troubleshooting techniques are outlined on docs.microsoft.com. ... View more