topic Re: Splunk search issues with timezone of logs from forwarders in Splunk Search

Splunk search issues with timezone of logs from forwarders

rchittip — Tue, 11 Jun 2019 15:04:14 GMT

Dears,

My Splunk Indexer is in CDT time zone and my forwarder logs are in UTC time zone and there is time difference of 5hrs. When I do the search in my splunk search head, data is getting indexed with 5 hour difference with the current time of splunk indexer.

Below are the forwarder logs:

2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/1111/000000/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 531 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/0910/2882183/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 515 
2019-06-11 12:50:42 10.100.4.65 GET /Test/GetStoreItemInv/2237/0544067/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 578 
2019-06-11 12:50:42 10.100.4.65 GET /ITest/GetStoreItemInv/2086/8513336/username/ - 9988 - 10.111.195.0 okhttp/2.6.0 - 200 0 0 671

I had updated the below stanza in on my forwarder /etc/system/loca/props.conf file but still nothing seems to be worked.

[ItmInqWebServiceWeb] 
TZ = America/Chicago

For time being, every time I search I'm adding "latest=+5h earliest=+45m" with my search.

Do I also need to update the above stanza in indexer server props.conf as well?

Thanks,
Ramu Chittiprolu

Re: Splunk search issues with timezone of logs from forwarders

schose — Tue, 11 Jun 2019 15:26:33 GMT

Hi,

TZ have to be set at parsing time - which means it will not work on universal forwarder. Set the setting on your indexers or intermediate heavy forwarders and it will fix you issue.

Best Regards,

Andreas

Re: Splunk search issues with timezone of logs from forwarders

harsmarvania57 — Tue, 11 Jun 2019 15:35:16 GMT

Are you running Forwarder on RedHat Linux ? If yes then is it RHEL 6 or RHEL 7 ?

Re: Splunk search issues with timezone of logs from forwarders

rchittip — Tue, 11 Jun 2019 16:12:04 GMT

Forwarder is on Windows server and splunk enterprise is on RHEL 6.1.

Re: Splunk search issues with timezone of logs from forwarders

harsmarvania57 — Tue, 11 Jun 2019 16:17:12 GMT

Have you tried with TZ=CDT on Forwarder ?

Re: Splunk search issues with timezone of logs from forwarders

rchittip — Tue, 11 Jun 2019 16:30:47 GMT

Yes, I tried below two in props.conf individually and restarted the forwarder but still search results are not correct.

[ItmInqWebServiceWeb]
TZ=CDT

[ItmInqWebServiceWeb]
TZ = America/Chicago

Re: Splunk search issues with timezone of logs from forwarders

harsmarvania57 — Tue, 11 Jun 2019 16:32:57 GMT

When you change timezone config on forwarder, it will apply to only new data. Data which is already ingested will not change with new timezone setting.

Re: Splunk search issues with timezone of logs from forwarders

rchittip — Tue, 11 Jun 2019 16:43:59 GMT

yes, I have the latest logs updated on the forwarder end but still no luck. Do I also need to update the TZ entry for sourcetype in indexer server as well ?

Re: Splunk search issues with timezone of logs from forwarders

harsmarvania57 — Wed, 12 Jun 2019 08:19:01 GMT

As far as I know, if you are running Forwarder and Indexer version 6.0+ then TZ on forwarder should work.

Re: Splunk search issues with timezone of logs from forwarders

rchittip — Wed, 12 Jun 2019 09:09:19 GMT

My forwarder and splunk version is 6.6.3. Not sure why this is not working.