Getting Data In

Why is there a 2 hour difference between _time and the actual events' timestamp for data on my receiver?

Explorer

I configured a 6.2 forwarder to send data to one of my receivers also running 6.2.
Data is getting into the receivers, but the problem is, the data which is being pulled into the receiver has a 2 hour delay.

Forwarder is in the Eastern Timezone and
Receiver is in Eastern Timezone.

I changed the parameter in ./system/local/
props.conf

[host::myhostname]
TZ = US/Eastern

and also configured in my app ./apps/appname/local/props.conf as

[host::recieverhostname]
TZ = America/Chicago
DATETIME_CONFIG = CURRENT

User Time zone is also set to Eastern Time (US & Canada).

However, I still see the data with a 2 hr delay in events, but _time is showing as Eastern Time.

For example:
_time is 8/17/16 11:57:20.000 AM
Event is "2016-08-17-09:57:20"

Now I want both _time and Event to be in sync. please suggest.

0 Karma

Motivator

Is this happening to all indexes or is it just some? If it is just some then are those indexes summary indexes?

If it is summary indexes then those events can get indexed according some different rules for timestamps. Basically Splunk uses the earliest portion of the time range. For example, if you have a search that populates an index with everything from that last 2 hours then everything will have a timestamp of 2 hours ago.

From the docs....

To set the time for summary index
events, Splunk software uses the
following information, in this order
of precedence:

  1. The _time value of the event being summarized.

  2. The earliest (or minimum) time of the scheduled search that populates
    the summary index. For example, if the
    summary-index-populating search covers
    the two minutes preceding each launch
    of its search, its earliest time is
    -2m.

  3. The current system time (in the case of an "all time" search, where no
    "earliest" value is specified)

In the majority of cases, your events
will have timestamps, so the first
method of discerning the summary index
timestamp holds. But if you are
summarizing data that doesn't contain
an _time field (such as data from a
lookup), the resulting events will
have the timestamp of the earliest
time of the summary-index-populating
search.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Usesummaryindexing#Summary_indexing_of_d...

0 Karma

Explorer

This is happening for normal indexes where the data is getting from the forwarder server.

For the indexes which are summary indexed the events are shown as:
Time - 8/19/16 12:30:03.000 PM
Event - 08/19/2016 12:30:00 -0400

If this is the case, then,
1. is there any thing we need to check from forwarder side ?
2. is Time Zone set for the user will effect?

0 Karma

Motivator

So summaries look pretty good.

What kind of data is the 2 hour difference?
Post the inputs.conf, props.conf and transforms.conf from the forwarder.

0 Karma

Communicator

Do you have a timezone in the raw logs e.g.

"2016-08-17-09:57:20 +02:00... "

OR

"2016-08-17-09:57:20 UTC ..."

0 Karma

Explorer

In the events data, when I selected as List, the datazone is shown as
date
zone = -360 ,
Time is shown as 8/19/16 12:01:32.000 PM
and
Event is shown as "2016-08-19 10:01:32"
-When selected as raw in dropdown, "2016-08-19 10:01:32" is shown

above shown for normal indexes
and
for indexes which was summary indexed is showing as
Time - 8/19/16 12:30:03.000 PM

Event - 08/19/2016 12:30:00 -0400,

0 Karma

Motivator

If everything else is in Eastern TZ then why do you have TZ = America/Chicago in your app? That is not the same timezone.

0 Karma

Explorer

Earlier the App was installed in other timezone so TZ parameter was set.

But, now the App(Receiver) and server(Forwarder) are at same location, why there is a difference of 2hrs?

0 Karma

Motivator

And the app could be overriding other settings. I would make it US/Eastern as a first step to see if that makes any difference. I can only assume that the data in question comes from that app, because you specifically mentioned it. (Eastern to Chicago is only 1 hour, but consistency can never hurt.)

Have you done a splunk btool props list --debug | grep "TZ" ?

0 Karma

Explorer

we tried that change US/Eastern, but not effected.
And no other parameter it is overriding.

0 Karma