How to set the time zone/alias for syslog data

[AhmadKhattak20]

Hi There,

So, the scenario is that we have a central syslog server which receives syslog messages from different servers in the organization. 

When the syslog data is received on the syslog server, there is a universal forwarder agent on the syslog server that forwards it to Splunk. The issue is that some servers are using UTC time zone and therefore the syslog data from those servers contains UTC Timestamp. 

Is there a way to change how the forwarder interprets the syslog data file received from those servers? I've tried editing TZ=UTC and TZ-ALIAS=UTC in props.conf with the source stanza specifying the path for those specific log files that have UTC Timestamp in events. However in Splunk I still see those events with the UTC Time Stamp. 

This is an issue due to which we can't properly search. Any advice would be much appreciated. Thanks.

[AhmadKhattak20]

Hi @richgalloway , yes there is a different directory and file maintained. Although I've not used the inputs.conf to set the TZ property rather I've used the props.conf to set the TZ property.

Hi @javiergn, Yes, I've used this source stanza in the props.conf file however the events from the file still show up with UTC Timestamp. 

[richgalloway]

You're right, the TZ property is set in props.conf, but it should be done on the UF.

[javiergn]

Can you send a screenshot of the event's raw and _time?

Simple run  your search:

index=foo sourcetype=bar source=yoursyslogsource
| head 1
| table _time, _raw

And also confirm what your user timezone is within the interface:

Regards,

J

[AhmadKhattak20]

Hi @javiergn ,

I ran the command that you gave me,

index=indexname sourcetype=syslog source=sourcename
| head 1
| _raw, _time

The result that I got was the following, (only showing the timestamps result)

_raw	_time
May 19 04:30:01	2021-05-19 04:30:01

It seems that Splunk is still extracting the time stamp from the event data itself and not converting the UTC time stamp to the one being used in Splunk Preferences. 

The time zone preference set in Splunk User Preferences is below

For reference what I've done in props.conf on the UF running on the Syslog Server is following,

[source::/path/*.log]
TZ=UTC
TZ_ALIAS=UTC

[javiergn]

Hi @AhmadKhattak20,

Couple of things:

- You don't need the TZ_ALIAS. TZ = UTC is perfectly valid

- Where is this props.conf located within your Splunk installation directory?

- Can you also paste the value of your source so that we can validate the stanza? Use the following query

index=indexname sourcetype=syslog source=sourcename
| head 1
| table _raw, _time, source

[AhmadKhattak20]

Hi @javiergn,

I've removed the TZ_ALIAS from the props.conf.

I'm pushing the app from the deployment server onto the Syslog Server where a Splunk UF is installed.

On the Syslog Server, this is located under /opt/splunk/etc/apps/custom-app-folder/local/props.conf

I ran the query that you mentioned and the results are following,

_raw	_time	source
May 19 09:10:01 ….	2021-05-19 09:10:01	/var/splunk/path/ipaddress/2021-05-19-servername.log

I verified that the source that was showing up in the query results is the same that I'm using in the props.conf stanza for source, (this props.conf is pushed on the syslog server - it is not present in any indexers/search head)

[source::/var/splunk/path/ipaddress/*.log]
TZ = UTC

[javiergn · Accepted solution]

OK, I think I know what the problem is thanks to your answer.

Your props.conf needs to go to your indexer (or Intermediate/Heavy Forwarder if there is any before reaching the Indexer). That's because you are using a Universal Forwarder and the TZ setting in props.conf is applied at parsing time:

https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

[AhmadKhattak20]

Thank you, I pushed the props.conf with the below stanza on indexers and now I'm getting expected results.

[source::/var/splunk/path/ipaddress/*.log]
TZ = UTC

[javiergn]

Great. Glad it worked.

Please don't forget to upvote the answers if you are happy with them.

Regards,

J

[richgalloway]

Have the syslog server put each source server's data into a different file.  The Universal Forwarder should monitor each file.  The inputs.conf file for the UF will have the appropriate TZ setting for each monitored file.

[javiergn]

Hi,

You can use the source:: and host:: stanzas within props.conf in order to edit the time zone for specific filepaths or host names. I normally use the source as it is unique to this particular type of data (Syslog) as opposed to the host. For instance:

[source::/var/log/syslog/firewall/myserverfilename*.log]
TZ=Europe/Madrid

See https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Regards,

J