topic Re: Custom timestamp detection for a sourcetype with some event w/o timestamp in Getting Data In

Custom timestamp detection for a sourcetype with some event w/o timestamp

D2SI — Wed, 08 Jan 2020 14:41:52 GMT

Hello there,

For a particular sourcetype there are events with a timestamp and events without timestamp.

As Splunk cannot detect a timestamp in the events without timestamp, it generates plenty of errors saying 'Defaulting to timestamp of previous event'.

I am trying to clean this up.

I see that I can configure a local datetime.xml to take into account various timestamp formats.

What I would like to do is :

for event with a timestamp -> assign the correct timestamp

for events without timestamp -> assign the indexing time as the timestamp and don't throw errors

Is there a way in datetime.xml to say : for this format, apply indextime timestamp?

Thanks in advance for any hint!

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

adonio — Wed, 08 Jan 2020 21:40:20 GMT

probably doesnt answer your question directly, but here is a workaround.
index all events with current timestamp
in search extract the timestamp for the field that contains time, lets say you will call it event_time
now in search (if need to use time other then current), run something like this:
... | eval _time = if(isnull(event_time),_time,event_time) ... or something along those lines
hope it helps a little bit

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

mloyola_splunk — Wed, 30 Sep 2020 03:36:34 GMT

I would suggest to use the big 6/8 of props.conf
Your props should contain these "TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD" and
add "DATETIME_CONFIG = current".

DATETIME_CONFIG is when Splunk can't find something to match TIME_* .

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

D2SI — Wed, 30 Sep 2020 03:33:18 GMT

Thanks for your answer.

Your are saying that with TIME_* parameters configured and DATETIME_CONFIG set to current, events with a timestamp will get timestamped according to TIME_* parameters while events from the same sourcetype without timestamp will have the indextime as the timestamp.

To my understanding setting DATETIME_CONFIG to current overrides all the TIME_* parameters and set the indextime as timestamp but I will test this.

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

adonio — Thu, 09 Jan 2020 13:22:34 GMT

would be very interesting, to know the result of your test @D2SI as i had the same understanding you had.
couple links for reference:
https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/Propsconf#Timestamp_extraction_configuration
https://wiki.splunk.com/Community:HowIndexingWorks

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

D2SI — Wed, 30 Sep 2020 03:33:47 GMT

Alright I have tested it and it does not work, I mean if DATETIME_CONFIG is set to CURRENT, it overrides any other TIME_* parameter and set timestamp to indextime.

Thanks anyhow for the suggestion.

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

jarizeloyola — Wed, 30 Sep 2020 03:37:37 GMT

Another way to solve this is to use INGEST_EVAL in transforms.conf.

I tried this config and works for me.

props.conf
[test]
LINE_BREAKER = ([\r\n]+)(\d+|\w+)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 30
TRANSFORMS = modtimestamp

transforms.conf

[modtimestamp]
INGEST_EVAL = _time[float]:=if(match(_raw,"^(?!\d+-\d+-\d+).*"),time(),_time)

it filters the events that doesnt start with the year and apply the condition

time() is more expensive and acts on every event separately. now() appears to only be valid in search and is the time of the search itself.

Note: This configuration might be very impactful on the indexer's performance.

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

adonio — Fri, 10 Jan 2020 14:10:53 GMT

imho the workaround i suggested in the comment directly under the question performs better.
and only being used if needed (search time)

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

jarizeloyola — Fri, 10 Jan 2020 14:26:49 GMT

Depends on his requirements , either at the index or search time. whatever that helps him. 🙂

Re: Custom timestamp detection for a sourcetype with some event w/o timestamp

D2SI — Wed, 30 Sep 2020 03:38:37 GMT

Thank both of you for your suggested solutions!

@adonio Good idea, thank you, I will test this with eval in the props.conf!

@jarizeloyola Thanks! I am using ingest_eval for another source which timestamp can only be found in the filename, and it works great! However, I see splunkd timestamp errors for that source, so I have to check if inges_eval works in a way there will still have timestamp issues generated even if OK, or if it is something else!