I configured a 6.2 forwarder to send data to one of my receivers also running 6.2.
Data is getting into the receivers, but the problem is, the data which is being pulled into the receiver has a 2 hour delay.
Forwarder is in the Eastern Timezone and
Receiver is in Eastern Timezone.
I changed the parameter in ./system/local/
props.conf
[host::myhostname]
TZ = US/Eastern
and also configured in my app ./apps/appname/local/props.conf
as
[host::recieverhostname]
TZ = America/Chicago
DATETIME_CONFIG = CURRENT
User Time zone is also set to Eastern Time (US & Canada).
However, I still see the data with a 2 hr delay in events, but _time is showing as Eastern Time.
For example:
_time is 8/17/16 11:57:20.000 AM
Event is "2016-08-17-09:57:20"
Now I want both _time and Event to be in sync. please suggest.
Is this happening to all indexes or is it just some? If it is just some then are those indexes summary indexes?
If it is summary indexes then those events can get indexed according some different rules for timestamps. Basically Splunk uses the earliest portion of the time range. For example, if you have a search that populates an index with everything from that last 2 hours then everything will have a timestamp of 2 hours ago.
From the docs....
To set the time for summary index
events, Splunk software uses the
following information, in this order
of precedence:
The _time value of the event being summarized.
The earliest (or minimum) time of the scheduled search that populates
the summary index. For example, if the
summary-index-populating search covers
the two minutes preceding each launch
of its search, its earliest time is
-2m.The current system time (in the case of an "all time" search, where no
"earliest" value is specified)In the majority of cases, your events
will have timestamps, so the first
method of discerning the summary index
timestamp holds. But if you are
summarizing data that doesn't contain
an _time field (such as data from a
lookup), the resulting events will
have the timestamp of the earliest
time of the summary-index-populating
search.
This is happening for normal indexes where the data is getting from the forwarder server.
For the indexes which are summary indexed the events are shown as:
Time - 8/19/16 12:30:03.000 PM
Event - 08/19/2016 12:30:00 -0400
If this is the case, then,
1. is there any thing we need to check from forwarder side ?
2. is Time Zone set for the user will effect?
So summaries look pretty good.
What kind of data is the 2 hour difference?
Post the inputs.conf
, props.conf
and transforms.conf
from the forwarder.
Do you have a timezone in the raw logs e.g.
"2016-08-17-09:57:20 +02:00... "
OR
"2016-08-17-09:57:20 UTC ..."
In the events data, when I selected as List, the data_zone is shown as
date_zone = -360 ,
Time is shown as 8/19/16 12:01:32.000 PM
and
Event is shown as "2016-08-19 10:01:32"
-When selected as raw in dropdown, "2016-08-19 10:01:32" is shown
above shown for normal indexes
and
for indexes which was summary indexed is showing as
Time - 8/19/16 12:30:03.000 PM
Event - 08/19/2016 12:30:00 -0400,
If everything else is in Eastern TZ then why do you have TZ = America/Chicago
in your app? That is not the same timezone.
Earlier the App was installed in other timezone so TZ parameter was set.
But, now the App(Receiver) and server(Forwarder) are at same location, why there is a difference of 2hrs?
And the app could be overriding other settings. I would make it US/Eastern as a first step to see if that makes any difference. I can only assume that the data in question comes from that app, because you specifically mentioned it. (Eastern to Chicago is only 1 hour, but consistency can never hurt.)
Have you done a splunk btool props list --debug | grep "TZ"
?
we tried that change US/Eastern, but not effected.
And no other parameter it is overriding.