Archive

How to adjust the time zone for an logs coming into splunk ?

Motivator

Hi All, Currently we are facing an issue time stamp for a firewall logs. We could see the logs are coming into splunk with a time difference of 3 hours.

Exact Problem:
Example : Current EDT time is 7:31 AM and logs are coming into splunk with a timestamp of
4:30:54.000 AM, so need to adjust the time zone by 3 hours to match the current EDT time.

inputs.conf detail :
[monitor:///opt/syslogs/mguard/.../mguard.log*]
index=fw
sourcetype=mguard:network:log
host_segment = 4

We have 5 heavy forwarder instance as intermediate forwarder and this firewall log is read from this 5 HF instance which is configured as syslogs server. The splunk reads the logs from these 5 HF instance and then ingest the data into indexer.

Kindly guide me how to adjust this time zone by 3 hours in splunk.

Tags (2)
0 Karma

SplunkTrust
SplunkTrust

In the props.conf you can adjust the TZ= setting for your sourcetype, you will need to do this on the heavy forwarder.
This is of course assuming the time is parsing as expected, if not consider configuring TIME_PREFIX and TIME_FORMAT in the props.conf file...

[firewall_sourcetype_goes_here]
TZ = GMT

Where you put the appropriate timezone/sourcetype above...there is more documentation around specifying time zones here

0 Karma

Motivator

hi Garethatiag, thanks for your effort, Currently we are getting the data with time stamp in EDT but only things time stamp has three hours behind the actual current EDT time.

Event details :

10/13/17
6:49:37.000 AM

Oct 13 06:49:37 XXXX.XXXX.com 1,2017/10/13 06:49:37,007257000034869,TRAFFIC,start,0,2017/10/13 06:49:37,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-XXXX,,,ssl,vsys1,trust,XXXX,ethernet1/2,ethernet1/1,Splunk,2017/10/13 06:49:37,751336,1,51214,10194,0,0,0x104041,tcp,allow,664,421,185,6,2017/10/13 06:49:37,0,computer-and-internet-info,0,6991231,0x0,x.0.0.0-x.255.255.255,United States,0,4,2,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.XXXXs.com source =/opt/syslogs/mguard/test01.XXXXs.com/mgaurd.log sourcetype = mguard:network:log
10/13/17
6:49:37.000 AM

Oct 13 06:49:37 test01.XXXXs.com 1,2017/10/13 06:49:37,007257000034869,TRAFFIC,s

Current time at EDT is 9:53 AM and if we can see there is 3 hours difference between them. so which i need to adjust to make it to the exact EDT time.

HF server time : Fri Oct 13 09:53:20 EDT 2017

Can I had props.conf like this and will this fix the timestamp issue.

[mguard:network:log]
TZ = EDT

0 Karma

Motivator

Hi All, Can anyone guide me on this issue.
thanks in advance.

0 Karma

SplunkTrust
SplunkTrust

Ok so the logs are timestamped as 3 hours behind what you expect, this shouldn't be too hard to fix.

Perhaps:
Etc/GMT+8

Or is it:

Etc/GMT+2

? As per this article the sign is inverted, the above is actually -08:00 and -02:00 in terms of time, which is 3 hours different from your current -05:00 timezone.

0 Karma

Motivator

hi garethatiag, thanks for working on this, could please let me know how to include above mentioned details in props.conf

Props.conf:
[mgaurd:network:log]
TZ = GMT+3

Below event detail are taken by keeping the time frame for last 24 hrs and current time in pennsylvania is 5:00 PM but index time is 3 hours behind the current time. So I need to fix this to match the current time.

Event details:

10/13/17
2:00:15.000 PM

Oct 13 14:00:15 test01.xxx.com1,2017/10/13 14:00:14,007257000034869,TRAFFIC,end,0,2017/10/13 14:00:14,10.x.x.x,51.x.x.x.x,0.0.0.0,0.0.0.0,trust-test01,,,incomplete,vsys1,trust,test01,ethernet1/2,ethernet1/1,Splunk,2017/10/13 14:00:14,770183,1,57307,443,0,0,0x4064,tcp,allow,132,132,0,2,2017/10/13 14:00:06,3,any,0,70039854,0x0,10.0.0.0-10.255.255.255,United States,0,2,0,aged-out,0,0,0,0,,test01,from-policy,,,0,,0,,N/A

thanks in advance.

0 Karma

Motivator

hi garethatiag kindly guide me on this issue, it has been taking my time since morning.

thanks in advance.

0 Karma

SplunkTrust
SplunkTrust

If you see the above comment, can you confirm you did restart/reindexed some data ?

When looking at the data that is an issue try adding this to the end:
| eval indextime=strftime(_indextime, "%+")

That will add an indextime to the fields available, the _time field will only be changed by props.conf for data indexed after you updated the props.conf / restarted the heavy forwarder.

0 Karma

Motivator

Hi garethatiag, I am did not understand what I need to do from the above comment. so please tell me what I need to add and where I need to add.

thanks in advance.

0 Karma

SplunkTrust
SplunkTrust

This was the comment that will change the time of new incoming data, test something like this by re-ingesting some data and see what happens, this is just props.conf so replace the other TZ= setting with this:

[mgaurd:network:log]
TZ = Etc/GMT+8

That will make the events appear to be 3 hours older than the EDT time of GMT-5 as per this article the +8 is actually -8 in time zones..
I think that will add 3 hours into the incoming events which is what you would expect but you will need to test it, I don't have access to a test instance at the moment.

0 Karma

Motivator

Hi garethatiag, I had tried the above stanza in props.conf but it did not work. Kindly guide me how to adjust this time zone by 3 hours in splunk.

thanks in advance

0 Karma

Motivator

Hi garethatiag, I had tried the below stanza in props.conf and it worked perfectly. Currently we could see log data are getting indexed as per the current time in EDT time zone. Thanks for your much need effort on this issue.

Props.conf
[mgaurd:network:log]
TZ = GMT

Now I could see the index time is matching the current time of EDT.

Motivator

Hi Garethatiag, Hey the issue is not fixed, we are facing same time stamp issue for firewall logs. Again the logs are coming into splunk with a time difference of 3 hours. Recently the firewall team has re-configured this device and the timezone on the device is now UTC . So I had updated the below stanza details in props.conf and after updating props.conf in the customized app , event data are not getting ingested into splunk.

[mgaurd:network:log]
TZ = UTC

Exact Two Problem:

1 )When the above the props.conf, is added into app, then the firewall data are not getting ingested into splunk.

2) Similarly when the above props.conf is removed from the customized app, then the firewall data are getting indexed into splunk but with a time difference of 3 hours.

Event details
10/17/17
4:21:56.000 AM
Oct 17 04:21:56 test01.xxx.com 1,2017/10/17 04:21:55,007257000034869,TRAFFIC,start,0,2017/10/17 04:21:55,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 04:21:55,229798,1,49472,10194,0,0,0x104041,tcp,allow,838,653,185,6,2017/10/17 04:21:55,0,computer-and-internet-info,0,70586295,0x0,10.x.x.x,10.x.x.x,United States,0,4,2,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in pennsylvania is 7:22 AM and if you can see the event data indexed time is 4:21 AM almost 3 hours difference its getting logged in.

Kindly guide me on this to fix the issue.

0 Karma

SplunkTrust
SplunkTrust

"1 )When the above the props.conf, is added into app, then the firewall data are not getting ingested into splunk."

Have you tested to see if that data went into the future?
Try searching with:
earliest=+5m latest=+5y

And see if the data appears, I'm wondering if the time went into the future.
If the timezone is definitely using UTC time, adjusting the TZ=UTC on either the heavy forwarder or the indexer receiving the data should work.
Do you receive the data on an indexer or heavy forwarder?

0 Karma

Motivator

Hi garethatiag, thanks for getting on this issue, Yes I had executed the below query to find out whether the data are appearing in future time but got no result found, after executing the query.
index=fw sourcetype=mguard:network:log earliest=+5m latest=+20y
| where _indextime < _time
| eval indextime=strftime(_indextime, "%+")

I had updated the props.conf in HF instance as the splunk sees the data from this location /opt/syslogs/mguard/test01/mguard.log then ingest the data into indexer instances.

If I add props.conf, firewall data are not getting ingested into splunk and if i remove the props.conf then data are getting indexed with 3 hour difference with the current time.

Kindly guide me how to troubleshoot this issue.

0 Karma

SplunkTrust
SplunkTrust

If the props.conf is an issue then splunk btool check should advise of an error.
If not the splunkd log file might drop a hint about what is happening here, perhaps check the splunkd log on the heavy forwarder ?

0 Karma

Motivator

thanks garethatiag, I had removed the props.conf app, so do you advise me put back the above props.conf stanza and check for btool and splunkd.log for an error. In that case what I need to search in splunkd.log ? i mean what key words need to use to filter out error related to this issue.

0 Karma

Motivator

Hi Garethatiag, After updating the below time zone in props.conf I could see time difference of 1 hour between the indexed time and the current time.

Props.conf
[mgaurd:network:log]
TZ = MST7MDT

Event data:

10/17/17
9:18:14.000 AM
Oct 17 07:18:14 test01.xxx.com 1,2017/10/17 07:18:14,007257000034869,TRAFFIC,start,0,2017/10/17 07:18:14,10.x.x.x,168.x.x.x,0.0.0.0,0.0.0.0,trust-xxxx,,,ssl,vsys1,trust,xxxx,ethernet1/2,ethernet1/1,Splunk,2017/10/17 07:18:14,238722,1,50351,10194,0,0,0x104041,tcp,allow,946,707,239,8,2017/10/17 07:18:14,0,computer-and-internet-info,0,70602352,0x0,10.x.x.x,10.x.x.x,United States,0,5,3,n/a,0,0,0,0,,test01,from-policy,,,0,,0,,N/A
host = test01.xxx.com source = /opt/syslogs/mguard/test01.xxx.com/mguard.log sourcetype = mguard:network:log

Current time in Pennsylvania is 10:18 AM .

difference of 1 hour between the indexed time and the current time.

kindly guide me to fix this .

0 Karma

Motivator

Hi Garethatiag, thanks for your effort and time on this issue, We have finally fixed this issue by updating the below stanza details in HF instances.

Props.conf
[mgaurd:network:log]
TZ = PST8PDT

Now we could see the index time is matching the current time of EDT.

0 Karma

SplunkTrust
SplunkTrust

Great, glad to hear you got it sorted...

0 Karma

SplunkTrust
SplunkTrust

Two things to confirm here, you are either pushing the props.conf via the deployment server to the heavy forwarder and it is triggering some kind of reload or restart.

Or you have restarted the heavy forwarder after updating the below props.conf ?
Finally, are you looking at newly indexed data?

A props.conf change related to timestamps will work for newly indexed data, the time of the already indexed events cannot be changed.

0 Karma