Splunk Search

Splunk search issues with timezone of logs from forwarders

Path Finder

Dears,

My Splunk Indexer is in CDT time zone and my forwarder logs are in UTC time zone and there is time difference of 5hrs. When I do the search in my splunk search head, data is getting indexed with 5 hour difference with the current time of splunk indexer.

Below are the forwarder logs:

2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/1111/000000/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 531 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/0910/2882183/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 515 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/2237/0544067/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 578 
2019-06-11 12:50:42 10.100.4.65 GET /ITest/GetStoreItemInv/2086/8513336/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 671 

I had updated the below stanza in on my forwarder /etc/system/loca/props.conf file but still nothing seems to be worked.

[ItmInqWebServiceWeb] 
TZ = America/Chicago 

For time being, every time I search I'm adding "latest=+5h earliest=+45m" with my search.

Do I also need to update the above stanza in indexer server props.conf as well?

Thanks,
Ramu Chittiprolu

0 Karma

SplunkTrust
SplunkTrust

Are you running Forwarder on RedHat Linux ? If yes then is it RHEL 6 or RHEL 7 ?

0 Karma

Path Finder

Forwarder is on Windows server and splunk enterprise is on RHEL 6.1.

0 Karma

SplunkTrust
SplunkTrust

Have you tried with TZ=CDT on Forwarder ?

0 Karma

Path Finder

Yes, I tried below two in props.conf individually and restarted the forwarder but still search results are not correct.

[ItmInqWebServiceWeb]
TZ=CDT

[ItmInqWebServiceWeb]
TZ = America/Chicago

0 Karma

SplunkTrust
SplunkTrust

When you change timezone config on forwarder, it will apply to only new data. Data which is already ingested will not change with new timezone setting.

0 Karma

Path Finder

yes, I have the latest logs updated on the forwarder end but still no luck. Do I also need to update the TZ entry for sourcetype in indexer server as well ?

0 Karma

SplunkTrust
SplunkTrust

As far as I know, if you are running Forwarder and Indexer version 6.0+ then TZ on forwarder should work.

0 Karma

Path Finder

My forwarder and splunk version is 6.6.3. Not sure why this is not working.

0 Karma

Contributor

Hi,

TZ have to be set at parsing time - which means it will not work on universal forwarder. Set the setting on your indexers or intermediate heavy forwarders and it will fix you issue.

Best Regards,

Andreas