Do not use last event timestamp for events without timestamp

[jeffland]

I have data which sometimes has timestamps and sometimes doesn't. I want those events without timestamp to use file mod time (it's a file monitor input), which is what the documentation leads me to believe is the default behavior if TIME_FORMAT doesn't match (https://docs.splunk.com/Documentation/Splunk/8.0.6/Data/HowSplunkextractstimestamps#How_Splunk_softw...).

However, I see my data sometimes matched to the last known timestamp instead, accompanied by these kind of messages in _internal:

WARN  DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event

How do I explicitly tell Splunk to not fall back to the previous timestamp and instead use file modification time for events without timestamps?

[richgalloway]

That documentation is a little misleading.  The file modification time is used if the first event does not have a timestamp.  After that, if  an event does not have a timestamp then the timestamp from the previous event is used.  Not sure that's documented anywhere, though, just my experience.

[jeffland]

So you're saying there is no way (you know of) to force splunk to change its behavior and it will always use the timestamp of the previous event?

[gjanders · Accepted solution]

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

[jeffland]

Well of course I can! Thanks for pointing this out, I don't know how I missed this option. Too focused on props.conf TIME_* settings I think.

I've used the following transforms for this:

INGEST_EVAL = _time := if(match(_raw, "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"), _time, now())

Only downside being that this is not as close to the actual event as file modification time would have been, as this happens on the indexer during parsing.

[gjanders]

From what I know there is no props.conf to fix this, so glad INGEST_EVAL helped.

I see timestamp mentioned in the docs but I'm not sure if that is a field.

What if you set the DATETIME_CONFIG = CURRENT

And then set the _time to the strptime() of the _raw or similar if it exists as _time in the ingest time eval?

[jeffland]

Even better. This uses file mod time for all events and selectively overwrites that value with the timestamp from the data if available. Nice thinking!