Hi,
With HEC token we see loss in logs.
1. Is there a way to get the logs that were lost?
2. How will we know that there are log loss?
1. Unless the data is saved outside of Splunk as well (as on a syslog server), there is no way to recover lost events.
2. How did you conclude you have loss to be able to say "With HEC token we see loss in logs"?
@richgalloway Customer informed us that some logs are missing and while checking we found at a particular timing certain logs has been missing.
The question is whether it was an error on the receiving side or the sending side. Of course, HEC is not that prone to log loss on its own as UDP syslog, but still, especially during downtime of one component or another, the loss can happen.
In my case (syslog to rsyslog to HEC), I can see in rsyslog logs if my downstream sending action failed and was suspended (and possibly failed over to another HEC endpoint). Do you have such info on your sending side?
Generally, if the event is received by HEC, unless there was some kind of a deep fault on splunk's behalf or you have some routing rules that filter out the events, the event should get indexed.
There is also one thing that can be confusing sometimes with HEC - the timestamp issues. If you're sending straight to /event endpoint, and you don't provide the timestamp with the event, one will not be parsed from the event contents unless you send with a particular parameter - the timestamp parsing step is completely bypassed within the parsing queue. So you might end up with different timestamp contained within the event and another one indexed in the _time field. Maybe that's your case - events are there just at another time.
Finding something that is not there is not Splunk's strong suit. The best we can do is monitor the amount of data coming in from various sources and alert when the volume changes by some amount/%. The TrackMe app can help with that.
Yes Trackme app works well with this situation and we have configured alerts whenever there is a delay or interruption in data ingestion