Dashboards & Visualizations

HEC token log loss

VijaySrrie
Builder

Hi,

With HEC token we see loss in logs.

1. Is there a way to get the logs that were lost?

2. How will we know that there are log loss?

 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

1. Unless the data is saved outside of Splunk as well (as on a syslog server), there is no way to recover lost events.

2. How did you conclude you have loss to be able to say "With HEC token we see loss in logs"?

---
If this reply helps you, Karma would be appreciated.
0 Karma

VijaySrrie
Builder

@richgalloway  Customer informed us that some logs are missing and while checking we found at a particular timing certain logs has been missing.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The question is whether it was an error on the receiving side or the sending side. Of course, HEC is not that prone to log loss on its own as UDP syslog, but still, especially during downtime of one component or another, the loss can happen.

In my case (syslog to rsyslog to HEC), I can see in rsyslog logs if my downstream sending action failed and was suspended (and possibly failed over to another HEC endpoint). Do you have such info on your sending side?

Generally, if the event is received by HEC, unless there was some kind of a deep fault on splunk's behalf or you have some routing rules that filter out the events, the event should get indexed.

There is also one thing that can be confusing sometimes with HEC - the timestamp issues. If you're sending straight to /event endpoint, and you don't provide the timestamp with the event, one will not be parsed from the event contents unless you send with a particular parameter - the timestamp parsing step is completely bypassed within the parsing queue. So you might end up with different timestamp contained within the event and another one indexed in the _time field. Maybe that's your case - events are there just at another time.

richgalloway
SplunkTrust
SplunkTrust

Finding something that is not there is not Splunk's strong suit.  The best we can do is monitor the amount of data coming in from various sources and alert when the volume changes by some amount/%.  The TrackMe app can help with that.

---
If this reply helps you, Karma would be appreciated.

Roy_9
Motivator

Yes Trackme app works well with this situation and we have configured alerts whenever there is a delay or interruption  in data ingestion

Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...