COnfused about TimeZones

[ocallender]

Hi folks, I've searched for an answer to this but haven't found anything that matches what I'm experiencing. For clarity, I am in Jamaica. We are in the EST time zone (GMT-5) but we do not observe daylight savings.

I started indexing device syslog messages forwarded from my Network Monitoring System via UDP:514. My NMS is on a windows server set to local time (GMT-5).

If an event occurs at 12:01pm Jamaica time (GMT-5), when I search for it in Splunk, that event has a _time of 7:01am (GMT-10). If I look at the syslog event viewer in my NMS, it shows the correct local time of 12:01pm (GMT-5).

Based on my research here, I learned that Splunk uses the time and zone of the server it is on if a timezone isn't specified in props.conf.

I checked the time on the server with the hwclock command at 2:08pm local time (GMT-5). The result: Fri 30 Aug 2013 08:54:43 AM EST

This got me confused. The hardware clock is set 5 hours in the past (Which is actually GMT-10) but has the timezone set to EST (GMT-5). How does that affect the way Splunk indexes events?

I'm guessing that I should do one of the following:
1. Change the hwclock on the Splunk server to the correct local time (GMT-5) and keep the timezone as EST.
2. Change the hwclock to the correct GMT time (GMT-0) and set the timezone to GMT.

What is considered the best practice for Splunk? Is it best to set the hardware clock to GMT or local time? How will this affect previously indexed items?

[ocallender · Accepted solution]

Update:
I set my hardware AND system clock to local time with EST timezone. Since then, the syslog events show the correct time stamps. However, previously indexed events didn't change, so I have a 5 hour gap in my events. I can live with that, as long as they're correct going forward.