How can we adjust our firewall's timezone?

[Hemnaath]

Hi All, Currently we are facing an issue with time stamp for an firewall logs. We could see the logs are coming into splunk with a time difference of 3 hours. We have 5 heavy forwarder instance as intermediate forwarder and this firewall log is read from this 5 HF instance which is configured as syslogs server. The splunk reads the logs from these 5 HF instance and then ingest the data into indexer.

inputs.conf detail :
[monitor:///opt/syslogs/mguard/.../mguard.log*]
index=fw
sourcetype=mguard:network:log
host_segment = 4

10/13/17
10:35:57.000 AM
Oct 13 10:35:57 test01.xxx.com 1,2017/10/13 10:35:57,007257000034869,TRAFFIC,start,0,2017/10/13 10:35:57,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/13 10:35:57,761997,1,51475,8089,0,0,0x104000,tcp,allow,416,350,66,4,2017/10/13 10:35:56,0,any,0,70021120,0x0,x.0.0.0-x.255.255.255,United States,0,3,1,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
eventtype = nix-all-logs eventtype = pan network host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log tag = network timeendpos = 16 timestartpos = 0

Current EDT time is 1:40 PM and logs are coming into splunk with a timestamp of
10:35:57.000 AM, so need to adjust the time zone by 3 hours to match the current EDT time.

Kindly guide me how to adjust this time zone by 3 hours in Splunk

[nickhills · Accepted solution]

Your firewall logs don't appear to specify a timezone offset, so Splunk will assume the timestamps are in UTC.

1.) Are you sure your firewall has the correct time?
2.) Does your firewall know what TZ its in?
3.) Can you amend your firewalls logs to include a TZ?
4.) Maybe you can "fix" this on the syslog server? - In my experience its always better to try and fix this as close to the source as possible.

[Hemnaath]

Hi Nickhillscpl, thanks for your effort on this. As I had commented earlier , both the HF instance and syslogs details are configured in the same node. and i had also included the TZ in props.conf file for the sourcetype mguard:network:log but it done fix the issue, we could see there is a difference of 3 hours between the current time and index time. Below props.conf has been deployed at HF instance from where the splunk reads the log and ingest into indexer.

Props.conf:
[mgaurd:network:log]
TZ = EDT

Kindly guide me how to adjust this time zone by 3 hours in Splunk

[agarrison]

http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Applytimezoneoffsetstotimestamps
This has some documentation.
Looks like you could try different zones listed in : /usr/share/zoneinfo

[nickhills]

I am not sure EDT is a valid TZ value, have you tried EST?
Did you restart the HF after updating the props file?

[nickhills]

actually - try "US/Eastern" or "EST5EDT" which (if works) will account for daylight saving

[Hemnaath]

hi nickhillspci, thanks for your effort on this, we have customized app and its pushed via deployer, in forwardmanagement we had mentioned enable the app /restart splunkd. so it should have been restarted when we execute splunk reload deploy-server.

Do I need to change the props.conf like this
[mgaurd:network:log]
TZ = EST5EDT

Below event detail are taken by keeping the time frame for last 24 hrs and current time in pennsylvania is 5:00 PM.
But in the event you can see the index time is 3 hours behind the current time. So I need to fix this to match the current time.

Event details:

10/13/17
2:00:15.000 PM
Oct 13 14:00:15 test01.xxx.com1,2017/10/13 14:00:14,007257000034869,TRAFFIC,end,0,2017/10/13 14:00:14,10.x.x.x,51.x.x.x.x,0.0.0.0,0.0.0.0,trust-test01,,,incomplete,vsys1,trust,test01,ethernet1/2,ethernet1/1,Splunk,2017/10/13 14:00:14,770183,1,57307,443,0,0,0x4064,tcp,allow,132,132,0,2,2017/10/13 14:00:06,3,any,0,70039854,0x0,10.0.0.0-10.255.255.255,United States,0,2,0,aged-out,0,0,0,0,,test01,from-policy,,,0,,0,,N/A

thanks in advance.

[Hemnaath]

Hi Nickhillspci, I had tried below props.conf stanza and it worked perfectly thank for your much need effort on this issue.

Props.conf
[mgaurd:network:log]
TZ = GMT

Now I could see the index time is matching the current time of EDT.

[nickhills]

Hi Hemnaath, that's great news. If this has solved the issue can you accept the answer so its marked as resolved.

[Hemnaath]

Hi Nickhillscpl, I have an issue now, data are not getting ingested into splunk from mguard logs, i am not sure whether it was happened due to above props.conf stanza. If that is not a case then kindly let me know what are trouble shooting steps should I need to follow to analysis the issue.

thanks in advance.

[Hemnaath]

Hi Nickhillscpl, Hey the issue is not fixed, we are facing same time stamp issue for firewall logs. Again the logs are coming into splunk with a time difference of 3 hours. The firewall team has re-configured this device and the timezone on the device is now UTC . So I had updated the below stanza details in props.conf and after updating props.conf in the customized app , event data are not getting ingested into splunk.

[mgaurd:network:log]
TZ = UTC

Event details
10/17/17
4:21:56.000 AM

Oct 17 04:21:56 test01.xxx.com 1,2017/10/17 04:21:55,007257000034869,TRAFFIC,start,0,2017/10/17 04:21:55,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 04:21:55,229798,1,49472,10194,0,0,0x104041,tcp,allow,838,653,185,6,2017/10/17 04:21:55,0,computer-and-internet-info,0,70586295,0x0,10.x.x.x,10.x.x.x,United States,0,4,2,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in pennsylvania is 7:22 AM and if you can see the event data indexed time is 4:21 AM almost 3 hours difference its getting logged in.

Exact Two Problem:

1 )When the above the props.conf, is added into app, then the firewall data are not getting ingested into splunk.
2) Similarly when the above props.conf is removed from the customized app, then the firewall data are getting indexed into splunk but with a time difference of 3 hours.

Kindly guide me on this to fix the issue.

[nickhills]

I am confused. - Lets work in UTC time only (no daylight saving)

Current UTC time as I type this is: 11:53
Your Firewall is now set to UTC? - Therefore the timestamps on your firewall should be 11:53?
However, your log indicates that the current time is 04:21 (which is 7 hours behind UTC)
You have imported the logs, and set the TZ in the props to tell splunk you are using UTC

In your user account settings - what is your splunk user's timezone set to?
Infact, what is your splunk servers timezone set to?

[nickhills]

EDIT: I removed my error from the last comment 🙂

Wait. I over thought that.
The numbers in my last comment are rubbish. But my two questions remain.

[Hemnaath]

Hi Nickhillscpl, thanks for getting into this problem, Yes I had checked the time set on both Heavy forwarder instance and indexer instances " Tue Oct 17 08:11:54 EDT 2017" and time zone for my user id in access control is defined as None. Kindly let me know where will be the issue now, not sure how to fix this issue.

[nickhills]

Ok, so to summarize (because I totally confused myself earlier 🙂 -
Your servers are in EDT, your browser is in EDT. So far so good.

Your firewall is sending events in (currently) UTC-7 So according to: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
You should try TZ = MST7MDT which will offset 7 hours, and account for DST

Out of interest - where is the firewall geographically located?

[Hemnaath]

thanks nickhillscpl, do you want me to update the props.conf stanza like this in HF instance.

Props.conf
[mgaurd:network:log]
TZ = MST7MDT

What it mean I did not understand this time zone .
Could not find the device location as the device appears to be on Azure cloud, by executing the tracert test01 from HF instances.

[nickhills]

https://simple.wikipedia.org/wiki/Mountain_Time_Zone

[Hemnaath]

hmm thanks so I will update the above props.conf stanza with the Mountain Time zone in HF instances. And validate whether its working or not.

[Hemnaath]

hi nickhillscpl thanks for your effort now I could see time difference of 1 hour between the indexed time and the current time.

Props.conf
[mgaurd:network:log]
TZ = MST7MDT

Event data:

10/17/17
9:18:14.000 AM

Oct 17 07:18:14 test01.xxx.com 1,2017/10/17 07:18:14,007257000034869,TRAFFIC,start,0,2017/10/17 07:18:14,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 07:18:14,238722,1,50351,10194,0,0,0x104041,tcp,allow,946,707,239,8,2017/10/17 07:18:14,0,computer-and-internet-info,0,70602352,0x0,10.x.x.x,10.x.x.x,United States,0,5,3,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in Pennsylvania is 10:18 AM .

difference of 1 hour between the indexed time and the current time.

kindly guide me to fix this .

[nickhills]

try TZ = PST8PDT - the starting point is "where" is the FW located

[sarwshai]

Hi, @nickhillspl I am having the same issue and all the forwarders which are lagging/leading are based on CEST timezone. so what should i put in TZ as i am unable to find a proper TZ from wikipedia link:https://en.wikipedia.org/wiki/List_of_tz_database_time_zones