Log files with different timezones (UTC)

[sonomauser]

I apologize since similar questions have been asked numerous times in the past. I have read several of them on this site as well as Splunk's own timezone article. I've tried a lot of things based on these articles, but the _time value doesn't appear to change at all.  I'm either doing something wrong or my expectations are off. 

Background:

We are PST. The Operating Systems for all our Splunk servers are configured for PST and are running Splunk 8.1.3. We are using a heavy forwarder to index IIS logs that are in UTC. 

When searching these logs in Splunk, I would like the canned times (Last 4 Hours, Last 60 Minutes, etc.) to reflect the PST-equivalent times they occurred. So if I'm searching for something that happened 30 minutes ago in real time, "Last 60 Minutes" will contain that log. 

It is my understanding that I am supposed to create/edit the props.conf on the heavy forwarder (/opt/splunk/etc/apps/iis/local/props.conf) and specify the TZ these logs files are set to:

[sourcetype_name]

TZ = UTC

Then restart Splunk on the heavy forwarder. 

This is done and I've restarted the entire Splunk farm. I've even set this in the /opt/splunk/etc/system/local/props.conf on the HF. These logs are still being indexed 7 hours into the future. 

Should this be working or am I thinking about this completely wrong? 

If my thinking is off-base, is it possible to accomplish what I'm attempting?

Any suggestions would be appreciated. 

Thank you. 

[PickleRick]

No.

YOU   SHOULD    NEVER     INTENTIONALLY    OFFSET     TIME    TO     WRONG    TIMEZONE !!!!!!

But seriously - the main goal of any log management or SIEM system is to provide you with the reliable version of time-anchored events.

The time the event happened should be parsed out from the event and set as an absolute time. If your hosts do work in PST and report the events in PST, you should parse them as being in PST and store them in splunk with an absolute timestamp which internally is just a number of seconds since epoch (there is no notion about any timezone here, mind you!).

It's up to the presentation layer (in this case - the splunk webui) to translate this timestamp to the proper timezone for the particular user. So if you have users from Europe, they can - if the want - show the times as CET, if you have users from New York, they can show the same timestamp as EST and so on.

You should never, never touch the timestamp!

[sonomauser]

Ok. Thank you.

I wasn't intentionally trying to alter the actual timestamp in the logs, I just wasn't quite sure how Splunk handled this type of thing. 

I was able to get it to work by putting TZ in the source type stanza on the Indexer, not the HF. I thought this had to be done on the HF, but I suppose that was my misunderstanding.

[PickleRick · Accepted solution]

OK. You're mixing several things 😉

One thing is the timestamp parsed from the event using timezone information according to

https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/Applytimezoneoffsetstotimestamps

I'd set the TZ in props for the particular source on HF

Another thing is how the time is rendered within the UI. The timezone here is configurable per user https://docs.splunk.com/Documentation/Splunk/8.2.3/Security/ConfigureuserswithSplunkWeb

[sonomauser]

Thank you again. 

I definitely read both of those articles before posting and took their suggestions, it just wasn't showing the PST time in Search and I was having a hard time conceptualizing how it should work or what I was doing wrong. 

It's working now, so thank you.