wrong timestamps in some files

[David_Hodgson]

I have a set of application logs, all processed into Splunk. Each has the entry timestam in the first 18 chars of the row. Each has its own sourcetype, and these are mostly identical in props.conf, apart from the name. Specifically they all have:

DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 18
TZ = UTC

One of the logfiles' timestamps are being correctly used and presented in Splunk as UTC. Three are presenting the timestamps as GMT (local time). When doing matching (e.g. joins) between logs they're failing as the timestamps appear different.

Here's an example of the defect:

The two rows are from different log files, one showing correct interpretation and the other showing incorrect.

Please advise:
1. what do I have to change so Splunk interprets ALL date-times as UTC
2. where should I look to find what's different between the sourcetypes / why they're producing different results?

I'm using Splunk desktop 6.4.2 on a Linux platform.
The files are read in from a set of watched folders, one for each sourcetype.

Thanks in advance for any assistance.
David

[tin_fish]

Hi David,

Yes not sure what happened there. I can't see my answer either. I've seen reports elsewhere of some preceding configurations overwriting the TZ set. Not the quickest form of resolution, but perhaps starting with your working props.conf and introducing other configurations not listed in step might reveal what is causing your problem?

(let's see if this answer sticks...).

[David_Hodgson]

Thanks for the help. I'll spend a few hours reloading and then we'll see. Consistently wrong I can live with 🙂

[horsefez]

Hi David,

the _time field does not neccesarely contain the time in your event field. Normaly the _time field gets populated when splunk indexes the events. If you want to alter the _time value you need to configure timestamp recognition manually.

Have a look into the following documents:
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps
http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

[David_Hodgson]

That seems to be the problem, unless I'm misunderstanding what's being presented to me by Splunk. The page you referred to says "Timestamp values are stored in the _time field (in UTC time format)." yet the log files which have the timezone explicitly set as UTC are showing an incorrect time in queries.

How do I stop Splunk from interpreting times into my local (desktop) timezone. This is corrupting the data.

[David_Hodgson]

... the list of timezones available for the user account doesn't include UTC.

[horsefez]

Hi David,

try to apply the following stanza in the props.conf.

[host::main]
TIME_FORMAT = %Y%m%d%H%M%S%3N

[David_Hodgson]

Here's a longer example covering all the log files:

[David_Hodgson]

I'm having a problem with the site, as the comment by tin_fish is no longer visible to me.

(S)he suggested I checked the sourcetype definition's and, sure enough, one was different: the log showing correct _times had a timezone of Auto and the others (with incorrect _times) had timezones of UTC.

I've corrected that (so they all say UTC) so they should all now be equally wrong, but will, at least, match in joins, transactions etc.

I've restarted the server, but it looks like I'll need to clean the index and reload the logs into the server ...

Any way to set the entire server so everything is UTC (including reports) so I don't need to worry about this again?

Any idea why _time is wrong when the sourcetype's TZ is right, and right when the TZ is Auto?