Getting Data In

Do not use last event timestamp for events without timestamp

jeffland
Champion
I have data which sometimes has timestamps and sometimes doesn't. I want those events without timestamp to use file mod time (it's a file monitor input), which is what the documentation leads me to believe is the default behavior if TIME_FORMAT doesn't match (https://docs.splunk.com/Documentation/Splunk/8.0.6/Data/HowSplunkextractstimestamps#How_Splunk_softw...).
 
However, I see my data sometimes matched to the last known timestamp instead, accompanied by these kind of messages in _internal:

 

WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event

 

How do I explicitly tell Splunk to not fall back to the previous timestamp and instead use file modification time for events without timestamps?
Labels (3)
1 Solution

gjanders
SplunkTrust
SplunkTrust

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That documentation is a little misleading.  The file modification time is used if the first event does not have a timestamp.  After that, if  an event does not have a timestamp then the timestamp from the previous event is used.  Not sure that's documented anywhere, though, just my experience.

---
If this reply helps you, an upvote would be appreciated.

jeffland
Champion

So you're saying there is no way (you know of) to force splunk to change its behavior and it will always use the timestamp of the previous event?

gjanders
SplunkTrust
SplunkTrust

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

jeffland
Champion

Well of course I can! Thanks for pointing this out, I don't know how I missed this option. Too focused on props.conf TIME_* settings I think.

I've used the following transforms for this:

INGEST_EVAL = _time := if(match(_raw, "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"), _time, now())

Only downside being that this is not as close to the actual event as file modification time would have been, as this happens on the indexer during parsing.

0 Karma

gjanders
SplunkTrust
SplunkTrust

From what I know there is no props.conf to fix this, so glad INGEST_EVAL helped.

I see timestamp mentioned in the docs but I'm not sure if that is a field.

What if you set the DATETIME_CONFIG = CURRENT

And then set the _time to the strptime() of the _raw or similar if it exists as _time in the ingest time eval?

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

jeffland
Champion

Even better. This uses file mod time for all events and selectively overwrites that value with the timestamp from the data if available. Nice thinking!

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!