Alerting

Anyone have a good alert to fire when data is injected by Splunk with a bad time stamp?

daniel333
Builder

All,

Say a log comes in dated 10 days older than today's date. I'd like a report or alert on that? Anyone have a good search for that handy?

0 Karma

woodcock
Esteemed Legend

There are several apps that help you dig into this, the 2 best are:
Meta Woot! https://splunkbase.splunk.com/app/2949/
Data Curator https://splunkbase.splunk.com/app/1848/

Splunk actually does a pretty good job of complaining about timestamp problems; it is just that most people do not look into it.

gjanders
SplunkTrust
SplunkTrust

Thanks for the link, data curator looks like it relates to the blog post linked above and meta-woot is a great app

0 Karma

gjanders
SplunkTrust
SplunkTrust

I have a few in Alerts for Splunk Admins or github the main one for your question would be:
IndexerLevel - Old data appearing in Splunk indexes

I also have:
IndexerLevel - Time format has changed multiple log types in one sourcetype
IndexerLevel - Valid Timestamp Invalid Parsed Time
IndexerLevel - Failures To Parse Timestamp Correctly (excluding breaking issues)
IndexerLevel - Future Dated Events that appeared in the last week
IndexerLevel - Too many events with the same timestamp

Among many others which may occur...

Note that in newer Splunk versions the data quality tab of the monitoring console will do most of the above.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@jkat54's comment is a good one. You might also want to look in index=_internal for log messages like "DateParserVerbose - Accepted time (Fri Aug 25 06:25:15 2017) is suspiciously far away from the previous event's time" and "DateParserVerbose - Failed to parse timestamp". They indicate potential problems with your timestamp extractions.

See http://runals.blogspot.com/2014/04/splunk-timestamps-and-dateparserverbose.html for a great discussion on the topic.

---
If this reply helps you, Karma would be appreciated.

gjanders
SplunkTrust
SplunkTrust

That's a nice query, will have to test the one on Mark's blog.

0 Karma

jkat54
SplunkTrust
SplunkTrust

How about this:

index=index 
| eval skew=_indextime-_time
| stats max(skew)  as max min(skew) as min avg(skew) as avg by sourcetype host 

_indextime is when it was indexed, _time is the time stamp extracted. It’s a starting point, from there you have to dig into the specific hosts and sourcetypes.

daniel333
Builder

Interesting! What is the unit in here? Seconds?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes, indeed

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...