topic Re: Do not use last event timestamp for events without timestamp in Getting Data In

Do not use last event timestamp for events without timestamp

jeffland — Thu, 01 Oct 2020 17:32:40 GMT

I have data which sometimes has timestamps and sometimes doesn't. I want those events without timestamp to use file mod time (it's a file monitor input), which is what the documentation leads me to believe is the default behavior if TIME_FORMAT doesn't match (https://docs.splunk.com/Documentation/Splunk/8.0.6/Data/HowSplunkextractstimestamps#How_Splunk_software_assigns_timestamps).

However, I see my data sometimes matched to the last known timestamp instead, accompanied by these kind of messages in _internal:

WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (23) characters of event. Defaulting to timestamp of previous event

How do I explicitly tell Splunk to not fall back to the previous timestamp and instead use file modification time for events without timestamps?

Re: Do not use last event timestamp for events without timestamp

richgalloway — Thu, 01 Oct 2020 19:13:42 GMT

That documentation is a little misleading. The file modification time is used if the first event does not have a timestamp. After that, if an event does not have a timestamp then the timestamp from the previous event is used. Not sure that's documented anywhere, though, just my experience.

Re: Do not use last event timestamp for events without timestamp

jeffland — Fri, 02 Oct 2020 06:26:31 GMT

So you're saying there is no way (you know of) to force splunk to change its behavior and it will always use the timestamp of the previous event?

Re: Do not use last event timestamp for events without timestamp

gjanders — Sat, 03 Oct 2020 00:08:22 GMT

Would an ingest time eval solve this?

Some examples here https://github.com/silkyrich/ingest_eval_examples

You might be able to set current time but I've never tried...

Re: Do not use last event timestamp for events without timestamp

jeffland — Mon, 05 Oct 2020 06:23:17 GMT

Well of course I can! Thanks for pointing this out, I don't know how I missed this option. Too focused on props.conf TIME_* settings I think.

I've used the following transforms for this:

INGEST_EVAL = _time := if(match(_raw, "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"), _time, now())

Only downside being that this is not as close to the actual event as file modification time would have been, as this happens on the indexer during parsing.

Re: Do not use last event timestamp for events without timestamp

gjanders — Mon, 05 Oct 2020 06:48:11 GMT

From what I know there is no props.conf to fix this, so glad INGEST_EVAL helped.

I see timestamp mentioned in the docs but I'm not sure if that is a field.

What if you set the DATETIME_CONFIG = CURRENT

And then set the _time to the strptime() of the _raw or similar if it exists as _time in the ingest time eval?

Re: Do not use last event timestamp for events without timestamp

jeffland — Mon, 05 Oct 2020 07:03:06 GMT

Even better. This uses file mod time for all events and selectively overwrites that value with the timestamp from the data if available. Nice thinking!