topic Re: manipulate timestamp in Splunk Search

manipulate timestamp

sarit_s — Tue, 11 Jun 2019 08:33:33 GMT

Hello
in my organisation we have few kinds of log format
one of them does not have the year in the time stamp so the event looks like:

Jun 6 02:32:43 : Info:Environment.cpp:27: MARINERVAR

this is causes me lots of problems in the report since splunk does not now what to do with this timestamp and i have cases where i get future time 😕

at the begging of the file i have full date
it looks like :

Thu Jun 6 02:32:43 CDT 2019

is it possible to use the year from the begging of the file and add it to timestamp at index time ?

thanks

Re: manipulate timestamp

DMohn — Tue, 11 Jun 2019 09:41:11 GMT

A missing year to your timestamp should not cause any problems if you have set up timestamp recognition in your props.conf correctly.

Try using the following parameters in props.conf for your relevant sourcetype (assuming the timestamp is at the beginning of your event):

[yoursourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 16

This should tell Splunk how to read your timestamp correctly and not produce any future-timestamped events, as it will try to stay as close to the current time as possible.

Re: manipulate timestamp

sarit_s — Tue, 11 Jun 2019 10:22:15 GMT

this is the configuration i have :

[fdm_f123_systemLog]
BREAK_ONLY_BEFORE = ^\w\s\d+\s\d{2}:\d{2}:\d{2}
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 15
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %b %e %H:%M:%S
TIME_PREFIX = ^
TRUNCATE = 0
category = Custom
disabled = false
pulldown_type = 1

is it ok ?

the problem is not only the future date
the problem is that it is possible that i will have events from 2018 at the same file
is it possible to take the year from somewhere else ?

Re: manipulate timestamp

DMohn — Tue, 11 Jun 2019 10:38:27 GMT

I am not 100% sure about this, but you can try to use an additional datetime.xml to extract the year from the filename. I am not aware of any method to exract the time (which is an index-time operation, hence done per-event) from any event earlier in the file.

Check https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configuredatetimexml for details of the datetime.xml usage.

Re: manipulate timestamp

sarit_s — Tue, 11 Jun 2019 10:44:59 GMT

taking it from file name will not help in that case since i can have events from year before

Re: manipulate timestamp

DMohn — Tue, 11 Jun 2019 12:34:46 GMT

In that case the only possibility would be - as bad as it sounds - to check your logging ...

If you get logs in one file that are years apart, I would personally consider the logging itself to be crap.

Re: manipulate timestamp

sarit_s — Wed, 12 Jun 2019 12:09:26 GMT

yeah i know.. it is not on my side
thanks