Splunk search issues with timezone of logs from forwarders

[rchittip]

Dears,

My Splunk Indexer is in CDT time zone and my forwarder logs are in UTC time zone and there is time difference of 5hrs. When I do the search in my splunk search head, data is getting indexed with 5 hour difference with the current time of splunk indexer.

Below are the forwarder logs:

2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/1111/000000/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 531 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/0910/2882183/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 515 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/2237/0544067/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 578 
2019-06-11 12:50:42 10.100.4.65 GET /ITest/GetStoreItemInv/2086/8513336/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 671 

I had updated the below stanza in on my forwarder /etc/system/loca/props.conf file but still nothing seems to be worked.

[ItmInqWebServiceWeb] 
TZ = America/Chicago 

For time being, every time I search I'm adding "latest=+5h earliest=+45m" with my search.

Do I also need to update the above stanza in indexer server props.conf as well?

Thanks,
Ramu Chittiprolu

[harsmarvania57]

Are you running Forwarder on RedHat Linux ? If yes then is it RHEL 6 or RHEL 7 ?

[rchittip]

Forwarder is on Windows server and splunk enterprise is on RHEL 6.1.

[harsmarvania57]

Have you tried with TZ=CDT on Forwarder ?

[rchittip]

Yes, I tried below two in props.conf individually and restarted the forwarder but still search results are not correct.

[ItmInqWebServiceWeb]
TZ=CDT

[ItmInqWebServiceWeb]
TZ = America/Chicago

[harsmarvania57]

When you change timezone config on forwarder, it will apply to only new data. Data which is already ingested will not change with new timezone setting.

[rchittip]

yes, I have the latest logs updated on the forwarder end but still no luck. Do I also need to update the TZ entry for sourcetype in indexer server as well ?

[harsmarvania57]

As far as I know, if you are running Forwarder and Indexer version 6.0+ then TZ on forwarder should work.

[rchittip]

My forwarder and splunk version is 6.6.3. Not sure why this is not working.

[schose]

Hi,

TZ have to be set at parsing time - which means it will not work on universal forwarder. Set the setting on your indexers or intermediate heavy forwarders and it will fix you issue.

Best Regards,

Andreas