The question is whether it was an error on the receiving side or the sending side. Of course, HEC is not that prone to log loss on its own as UDP syslog, but still, especially during downtime of one component or another, the loss can happen. In my case (syslog to rsyslog to HEC), I can see in rsyslog logs if my downstream sending action failed and was suspended (and possibly failed over to another HEC endpoint). Do you have such info on your sending side? Generally, if the event is received by HEC, unless there was some kind of a deep fault on splunk's behalf or you have some routing rules that filter out the events, the event should get indexed. There is also one thing that can be confusing sometimes with HEC - the timestamp issues. If you're sending straight to /event endpoint, and you don't provide the timestamp with the event, one will not be parsed from the event contents unless you send with a particular parameter - the timestamp parsing step is completely bypassed within the parsing queue. So you might end up with different timestamp contained within the event and another one indexed in the _time field. Maybe that's your case - events are there just at another time.
... View more