Splunk Search

manipulate timestamp

sarit_s
Communicator

Hello
in my organisation we have few kinds of log format
one of them does not have the year in the time stamp so the event looks like:

Jun 6 02:32:43 : Info:Environment.cpp:27: MARINERVAR

this is causes me lots of problems in the report since splunk does not now what to do with this timestamp and i have cases where i get future time 😕

at the begging of the file i have full date
it looks like :

Thu Jun 6 02:32:43 CDT 2019

is it possible to use the year from the begging of the file and add it to timestamp at index time ?

thanks

Tags (1)
0 Karma

DMohn
Motivator

A missing year to your timestamp should not cause any problems if you have set up timestamp recognition in your props.conf correctly.

Try using the following parameters in props.conf for your relevant sourcetype (assuming the timestamp is at the beginning of your event):

[yoursourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 16

This should tell Splunk how to read your timestamp correctly and not produce any future-timestamped events, as it will try to stay as close to the current time as possible.

0 Karma

sarit_s
Communicator

this is the configuration i have :

[fdm_f123_systemLog]
BREAK_ONLY_BEFORE = ^\w\s\d+\s\d{2}:\d{2}:\d{2}
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 15
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %b %e %H:%M:%S
TIME_PREFIX = ^
TRUNCATE = 0
category = Custom
disabled = false
pulldown_type = 1

is it ok ?

the problem is not only the future date
the problem is that it is possible that i will have events from 2018 at the same file
is it possible to take the year from somewhere else ?

0 Karma

DMohn
Motivator

I am not 100% sure about this, but you can try to use an additional datetime.xml to extract the year from the filename. I am not aware of any method to exract the time (which is an index-time operation, hence done per-event) from any event earlier in the file.

Check https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configuredatetimexml for details of the datetime.xml usage.

0 Karma

sarit_s
Communicator

taking it from file name will not help in that case since i can have events from year before

0 Karma

DMohn
Motivator

In that case the only possibility would be - as bad as it sounds - to check your logging ...

If you get logs in one file that are years apart, I would personally consider the logging itself to be crap.

0 Karma

sarit_s
Communicator

yeah i know.. it is not on my side
thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...