Getting Data In

Do not use last event timestamp for events without timestamp

jeffland
Champion
I have data which sometimes has timestamps and sometimes doesn't. I want those events without timestamp to use file mod time (it's a file monitor input), which is what the documentation leads me to believe is the default behavior if TIME_FORMAT doesn't match (https://docs.splunk.com/Documentation/Splunk/8.0.6/Data/HowSplunkextractstimestamps#How_Splunk_softw...).
 
However, I see my data sometimes matched to the last known timestamp instead, accompanied by these kind of messages in _internal:

 

WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event

 

How do I explicitly tell Splunk to not fall back to the previous timestamp and instead use file modification time for events without timestamps?
Labels (3)
1 Solution

gjanders
SplunkTrust
SplunkTrust

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That documentation is a little misleading.  The file modification time is used if the first event does not have a timestamp.  After that, if  an event does not have a timestamp then the timestamp from the previous event is used.  Not sure that's documented anywhere, though, just my experience.

---
If this reply helps you, Karma would be appreciated.

jeffland
Champion

So you're saying there is no way (you know of) to force splunk to change its behavior and it will always use the timestamp of the previous event?

gjanders
SplunkTrust
SplunkTrust

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

0 Karma

jeffland
Champion

Well of course I can! Thanks for pointing this out, I don't know how I missed this option. Too focused on props.conf TIME_* settings I think.

I've used the following transforms for this:

INGEST_EVAL = _time := if(match(_raw, "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"), _time, now())

Only downside being that this is not as close to the actual event as file modification time would have been, as this happens on the indexer during parsing.

0 Karma

gjanders
SplunkTrust
SplunkTrust

From what I know there is no props.conf to fix this, so glad INGEST_EVAL helped.

I see timestamp mentioned in the docs but I'm not sure if that is a field.

What if you set the DATETIME_CONFIG = CURRENT

And then set the _time to the strptime() of the _raw or similar if it exists as _time in the ingest time eval?

jeffland
Champion

Even better. This uses file mod time for all events and selectively overwrites that value with the timestamp from the data if available. Nice thinking!

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...