Set timezone in universal forwarder

[kapilbk1996]

Machine A(universal forwarder)forwards logs to machine B(indexer). Dummy logs that are being forwarded by machine A are shown in "pic_1". The "time" field for each event denotes the time in GMT.
Also, props.log at forwarder is shown in "pic_2".
The issue is that I specified the time zone as IST but the events get GMT based timestamp.
Moreover when I try to perform search operation for relative time frame such as '2 min ago', It shows - 'No record found'. (Pic_3) even though records are present in that time frame.
Kindly help me out with the same.

Pic1:
https://drive.google.com/file/d/1EZA46tTRD0ffdo2stLtyINhgd_ABrElz/view?usp=drivesdk

Pic2:
https://drive.google.com/file/d/1b66cxFbHPcmwJQm1jeaIW_t3hfPvyW8k/view?usp=drivesdk

Pic3:
https://drive.google.com/file/d/1KUax2BRk9HDpDnTpOfv-s1eGOL6nYZIn/view?usp=drivesdk

[GabrielTarget · Accepted solution]

Howdy Kapilbk!

From what I could see in your pics these are my recommendations for your issue:

First your Time Zone must be one that Linux knows, here's this wikipedia link that has a list of which TZ could you use:

https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

If you want to specify the timezone that India uses (UTC+5:30), you could add in your props.conf under the [default] stanza the following line:

TZ = Asia/Kolkata

If you want to read more about it, here's this Splunk Doc:

http://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Applytimezoneoffsetstotimestamps

Now, about the events that doesn't appear in pic_3, from what I could see in your pic_1 the lastest event was around 1 PM (13 h), and seeing that the server were Splunk was at 7 PM at the moment, looking for the last 2 minutes will not shown them. This could be also be related to your timezone trouble.

I hope this solves your issue!

[kapilbk1996]

Thanks for the response.

My events are indexed with GMT timestamp and the search operation uses IST timezone to search records.
Let us suppose that a record 'X' is added at 4:36 AM (GMT). Now when I perform search operation from IST timezone (10:06 AM IST) for relative time frame of "2 min ago", it will search for time frame of 10:04 to 10:06 AM, but the record 'X' (that was added at 10:06 AM IST) does not show up. This is because record 'X' got the timestamp of '4:36 AM GST'. But according to IST timezone, it occurred at 10:06 AM IST and should have been shown in the output.

All I want is that even though my events are indexed in GMT, but my search should work according to IST time zone. What changes are needed to be done to achieve the same?

[GabrielTarget]

Howdy again Kapilbk!

This could be happening because the you have yet to add the timezone to your props.conf file, the reason I came to this conclusion is the following:

You have a log A, let's name it "Log_from_India", whose events will be all in IST timezone, and a log B "Log_from_Brasil", which will have a timezone of GMT-3, in both scenarios you use the same sourcetype "log4j" (From what I saw in pic_1) that uses the Auto Extraction of timestamp by default, if you don't specify to Splunk what timezone do they come from, it will use the timezone of the local server where Splunk resides, so both Log_from_Brasil and Log_from_India will be indexed with the same timezone (In your case IST).

So returning to your example, even if your logs are from a place where the timezone is GMT, if you don't add any configuration to the sourcetype nor the default stanza that specify this, both events will be indexed as your local timezone, so your 4:36 AM GMT will be seen in Splunk as 4:36 AM IST and not 9:30 AM IST.

So if your logs from an Universal Forwarder come from a timezone that is different of your own, add in the sourcetype stanza the line TZ = TIMEZONE_FROM_YOUR_UF, for example:

TZ = Etc/GMT

I highly recommend as a good practice to create a new sourcetype where you can add this configuration, adding lines to the default stanza of props.conf can bring issues later on.

Regards!