Timestamp problem

[dhs_harry08]

Hi,

I have scheduled a cron in unix for every one minute which takes the uptime command output and writes to a file. It has been over 4 weeks now and I am observing now that every day splunk timestamp changes abruptly at 16:29.

You can see from below logs the event after 2013-05-27T16:29:00.000+0530 is coming as 2013-05-28T04:30:00.000+0530 and because of this a lot of events are missing and not getting reported.

1:00pm up 146 days 1:10, 1 user, load average: 2.11, 2.27, 3.20 2013-05-28T04:30:00.000+0530 1 28 0 may tuesday 2013 local nix-all-logs byk300lin241 3.2 2.11 2.27 /home/tkaushal/load.log unix_load 6 2

12:59am up 145 days 13:09, 0 users, load average: 0.24, 0.84, 1.15 2013-05-27T16:29:00.000+0530 12 27 59 may monday 2013 local nix-all-logs byk300lin241 1.15 0.24 0.84 /home/tkaushal/load.log unix_load 6 1

Please help me understand why is this ocuring and solution for it.

Regards,
Harish

[okrabbe_splunk · Accepted solution]

Harish,

You should try something like this in props.conf

[mysourcetype]
MAX_TIMESTAMP_LOOKAHEAD=100
TIME_PREFIX = ^(\S+\s+){12}
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

You could also use a source stanza (your log file name) instead but sourcetype is really a better way to do it.

I did not test this but it should be close. The link I posted earlier goes into a lot of detail around this topic:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

[dhs_harry08]

I will try this out. Thanks okrabbe

[bmacias84]

I see your event has two timestamps within the data, 1:00pm and 2013-05-28T04:30:00.000+0530. Splunk can sometimes get confused if you are using automatic timestamp recognition. I recommend configuring timestamp reconnition MAX_TIMESTAMP_LOOKAHEAD, TIME_PREFIX, TIME_FORMAT for this source. Configuring these can increase index peformance.

[dhs_harry08]

Yes I am relying on automatic timestamp recognition. But this is happening for only one sourcetype. All the others are fine.
If I change the timestamp will it not affect the others. Also is that the only option I got to correct it.

[okrabbe_splunk]

Are you relying on Splunk's automatic timestamp recognition?

If so, it is often more efficient and you can be certain it works correctly if you manually specify the timestamp recognition in props.conf. It should only take a few minutes.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition