Getting Data In

How can we adjust our firewall's timezone?

Hemnaath
Motivator

Hi All, Currently we are facing an issue with time stamp for an firewall logs. We could see the logs are coming into splunk with a time difference of 3 hours. We have 5 heavy forwarder instance as intermediate forwarder and this firewall log is read from this 5 HF instance which is configured as syslogs server. The splunk reads the logs from these 5 HF instance and then ingest the data into indexer.

inputs.conf detail :
[monitor:///opt/syslogs/mguard/.../mguard.log*]
index=fw
sourcetype=mguard:network:log
host_segment = 4

10/13/17
10:35:57.000 AM
Oct 13 10:35:57 test01.xxx.com 1,2017/10/13 10:35:57,007257000034869,TRAFFIC,start,0,2017/10/13 10:35:57,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/13 10:35:57,761997,1,51475,8089,0,0,0x104000,tcp,allow,416,350,66,4,2017/10/13 10:35:56,0,any,0,70021120,0x0,x.0.0.0-x.255.255.255,United States,0,3,1,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
eventtype = nix-all-logs eventtype = pan network host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log tag = network timeendpos = 16 timestartpos = 0

Current EDT time is 1:40 PM and logs are coming into splunk with a timestamp of
10:35:57.000 AM, so need to adjust the time zone by 3 hours to match the current EDT time.

Kindly guide me how to adjust this time zone by 3 hours in Splunk

0 Karma
1 Solution

nickhills
Ultra Champion

Your firewall logs don't appear to specify a timezone offset, so Splunk will assume the timestamps are in UTC.

1.) Are you sure your firewall has the correct time?
2.) Does your firewall know what TZ its in?
3.) Can you amend your firewalls logs to include a TZ?
4.) Maybe you can "fix" this on the syslog server? - In my experience its always better to try and fix this as close to the source as possible.

If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

Your firewall logs don't appear to specify a timezone offset, so Splunk will assume the timestamps are in UTC.

1.) Are you sure your firewall has the correct time?
2.) Does your firewall know what TZ its in?
3.) Can you amend your firewalls logs to include a TZ?
4.) Maybe you can "fix" this on the syslog server? - In my experience its always better to try and fix this as close to the source as possible.

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

Hi Nickhillscpl, thanks for your effort on this. As I had commented earlier , both the HF instance and syslogs details are configured in the same node. and i had also included the TZ in props.conf file for the sourcetype mguard:network:log but it done fix the issue, we could see there is a difference of 3 hours between the current time and index time. Below props.conf has been deployed at HF instance from where the splunk reads the log and ingest into indexer.

Props.conf:
[mgaurd:network:log]
TZ = EDT

Kindly guide me how to adjust this time zone by 3 hours in Splunk

0 Karma

agarrison
Path Finder

http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Applytimezoneoffsetstotimestamps
This has some documentation.
Looks like you could try different zones listed in : /usr/share/zoneinfo

0 Karma

nickhills
Ultra Champion

I am not sure EDT is a valid TZ value, have you tried EST?
Did you restart the HF after updating the props file?

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

actually - try "US/Eastern" or "EST5EDT" which (if works) will account for daylight saving

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

hi nickhillspci, thanks for your effort on this, we have customized app and its pushed via deployer, in forwardmanagement we had mentioned enable the app /restart splunkd. so it should have been restarted when we execute splunk reload deploy-server.

Do I need to change the props.conf like this
[mgaurd:network:log]
TZ = EST5EDT

Below event detail are taken by keeping the time frame for last 24 hrs and current time in pennsylvania is 5:00 PM.
But in the event you can see the index time is 3 hours behind the current time. So I need to fix this to match the current time.

Event details:

10/13/17
2:00:15.000 PM
Oct 13 14:00:15 test01.xxx.com1,2017/10/13 14:00:14,007257000034869,TRAFFIC,end,0,2017/10/13 14:00:14,10.x.x.x,51.x.x.x.x,0.0.0.0,0.0.0.0,trust-test01,,,incomplete,vsys1,trust,test01,ethernet1/2,ethernet1/1,Splunk,2017/10/13 14:00:14,770183,1,57307,443,0,0,0x4064,tcp,allow,132,132,0,2,2017/10/13 14:00:06,3,any,0,70039854,0x0,10.0.0.0-10.255.255.255,United States,0,2,0,aged-out,0,0,0,0,,test01,from-policy,,,0,,0,,N/A

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Nickhillspci, I had tried below props.conf stanza and it worked perfectly thank for your much need effort on this issue.

Props.conf
[mgaurd:network:log]
TZ = GMT

Now I could see the index time is matching the current time of EDT.

0 Karma

nickhills
Ultra Champion

Hi Hemnaath, that's great news. If this has solved the issue can you accept the answer so its marked as resolved.

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

Hi Nickhillscpl, I have an issue now, data are not getting ingested into splunk from mguard logs, i am not sure whether it was happened due to above props.conf stanza. If that is not a case then kindly let me know what are trouble shooting steps should I need to follow to analysis the issue.

thanks in advance.

0 Karma

Hemnaath
Motivator

Hi Nickhillscpl, Hey the issue is not fixed, we are facing same time stamp issue for firewall logs. Again the logs are coming into splunk with a time difference of 3 hours. The firewall team has re-configured this device and the timezone on the device is now UTC . So I had updated the below stanza details in props.conf and after updating props.conf in the customized app , event data are not getting ingested into splunk.

[mgaurd:network:log]
TZ = UTC

Event details
10/17/17
4:21:56.000 AM

Oct 17 04:21:56 test01.xxx.com 1,2017/10/17 04:21:55,007257000034869,TRAFFIC,start,0,2017/10/17 04:21:55,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 04:21:55,229798,1,49472,10194,0,0,0x104041,tcp,allow,838,653,185,6,2017/10/17 04:21:55,0,computer-and-internet-info,0,70586295,0x0,10.x.x.x,10.x.x.x,United States,0,4,2,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in pennsylvania is 7:22 AM and if you can see the event data indexed time is 4:21 AM almost 3 hours difference its getting logged in.

Exact Two Problem:

1 )When the above the props.conf, is added into app, then the firewall data are not getting ingested into splunk.
2) Similarly when the above props.conf is removed from the customized app, then the firewall data are getting indexed into splunk but with a time difference of 3 hours.

Kindly guide me on this to fix the issue.

0 Karma

nickhills
Ultra Champion

I am confused. - Lets work in UTC time only (no daylight saving)

Current UTC time as I type this is: 11:53
Your Firewall is now set to UTC? - Therefore the timestamps on your firewall should be 11:53?
However, your log indicates that the current time is 04:21 (which is 7 hours behind UTC)
You have imported the logs, and set the TZ in the props to tell splunk you are using UTC

In your user account settings - what is your splunk user's timezone set to?
Infact, what is your splunk servers timezone set to?

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

EDIT: I removed my error from the last comment 🙂

Wait. I over thought that.
The numbers in my last comment are rubbish. But my two questions remain.

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

Hi Nickhillscpl, thanks for getting into this problem, Yes I had checked the time set on both Heavy forwarder instance and indexer instances " Tue Oct 17 08:11:54 EDT 2017" and time zone for my user id in access control is defined as None. Kindly let me know where will be the issue now, not sure how to fix this issue.

0 Karma

nickhills
Ultra Champion

Ok, so to summarize (because I totally confused myself earlier 🙂 -
Your servers are in EDT, your browser is in EDT. So far so good.

Your firewall is sending events in (currently) UTC-7 So according to: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
You should try TZ = MST7MDT which will offset 7 hours, and account for DST

Out of interest - where is the firewall geographically located?

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

thanks nickhillscpl, do you want me to update the props.conf stanza like this in HF instance.

Props.conf
[mgaurd:network:log]
TZ = MST7MDT

What it mean I did not understand this time zone .
Could not find the device location as the device appears to be on Azure cloud, by executing the tracert test01 from HF instances.

0 Karma

nickhills
Ultra Champion

https://simple.wikipedia.org/wiki/Mountain_Time_Zone

If my comment helps, please give it a thumbs up!
0 Karma

Hemnaath
Motivator

hmm thanks so I will update the above props.conf stanza with the Mountain Time zone in HF instances. And validate whether its working or not.

0 Karma

Hemnaath
Motivator

hi nickhillscpl thanks for your effort now I could see time difference of 1 hour between the indexed time and the current time.

Props.conf
[mgaurd:network:log]
TZ = MST7MDT

Event data:

10/17/17
9:18:14.000 AM

Oct 17 07:18:14 test01.xxx.com 1,2017/10/17 07:18:14,007257000034869,TRAFFIC,start,0,2017/10/17 07:18:14,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 07:18:14,238722,1,50351,10194,0,0,0x104041,tcp,allow,946,707,239,8,2017/10/17 07:18:14,0,computer-and-internet-info,0,70602352,0x0,10.x.x.x,10.x.x.x,United States,0,5,3,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in Pennsylvania is 10:18 AM .

difference of 1 hour between the indexed time and the current time.

kindly guide me to fix this .

0 Karma

nickhills
Ultra Champion

try TZ = PST8PDT - the starting point is "where" is the FW located

If my comment helps, please give it a thumbs up!
0 Karma

sarwshai
Communicator

Hi, @nickhillspl I am having the same issue and all the forwarders which are lagging/leading are based on CEST timezone. so what should i put in TZ as i am unable to find a proper TZ from wikipedia link:https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...