topic Re: How to fix wrong timestamping issues? in Getting Data In

How to fix wrong timestamping issues?

rangineniarunku — Fri, 24 Mar 2017 22:02:43 GMT

I missed some of the events for my search query, when I try to evaluate the time diff between event time and index time, Index time is early than event time. How can I resolve the timing issue? Where should I navigate and check??

index=Bullseye sourcetype=BullseyeAppLogs source="\\\\BP1XTXII492\\BULLSEYE.logs.devc$\\1817*.log" | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") | eval diff= _indextime-_time | table _time, indextime, diff

output:

_time↕ indextime↕ diff↕

2017-03-22 18:56:13 2017-03-22 19:01:48 335

2017-03-22 18:55:45 2017-03-22 19:01:48 363

2017-03-22 18:44:03 2017-03-24 18:28:29 171866

2017-03-22 18:43:27 2017-03-24 18:28:29 171902

2017-03-22 18:42:41 2017-03-22 18:44:57 136

2017-03-22 18:42:26 2017-03-22 18:44:57 151

2017-03-22 18:34:35 2017-03-22 18:35:09 34

2017-03-22 18:33:55 2017-03-22 18:35:09 74

2017-03-22 18:33:48 2017-03-24 18:28:02 172454

2017-03-22 18:33:14 2017-03-24 18:28:02 172488

2017-03-22 18:31:09 2017-03-24 18:28:24 172635

2017-03-22 18:30:46 2017-03-24 18:27:15 172589

2017-03-22 18:30:42 2017-03-24 18:28:24 172662

2017-03-22 18:30:35 2017-03-24 18:27:10 172595

2017-03-22 18:29:48 2017-03-24 18:27:15 172647

2017-03-22 18:29:02 2017-03-24 18:27:10 172688

2017-03-22 18:19:43 2017-03-24 18:28:21 173318

2017-03-22 18:19:11 2017-03-24 18:28:21 173350

2017-03-22 18:16:57 2017-03-22 18:18:02 65

2017-03-22 18:16:24 2017-03-22 18:18:02 98

Re: How to fix wrong timestamping issues?

gjanders — Sat, 25 Mar 2017 04:59:28 GMT

Assuming your logs are not printing future-based date/time stamps within them, then perhaps Splunk is not correctly parsing the date/time.

I would suggest you refer to Configure event timestamps , in particular the link to Configure timestamp recognition .
Just make sure you configure the timestamp recognition at the heavy forwarder or indexer level where the data is sent to.

For most sourcetypes I configure time parsing by using:

TIME_PREFIX = 
TIME_FORMAT =

I also sometimes configure:

MAX_TIMESTAMP_LOOKAHEAD = 
TZ =

When required.

Re: How to fix wrong timestamping issues?

woodcock — Tue, 28 Mar 2017 23:59:20 GMT

If _indextime is earlier than _time, then one of the following is true:

The timestamping entity (the process writing to the file on the forwarder) is writing the wrong time. This is usually because the host OS has the wrong time. This is usually because the admin of that device neglected to setup NTP.

The timestamping entity is using a proper TZ but is not writing this TZ in the timestamp. This can be fixed by using the TZ=feature and deploying this to the forwarder (if post 6.0) or the Indexers (if pre-6.0).

The timestamping entity is using the wrongTZ and is writing this TZ in the timestamp. In the short-term, this can be fixed using the TZ_ALIAS= feature, but the proper fix is to correct the timestamping entity's insanity.

You have not correctly told Splunk where to find the timestamp (DO NOT EVER allow it to automatically timestamp: tell it exactly where and in what format the timestamp is) so it is mistaking non-timestamp data as a timestamp, misinterpreting the timestamp, or giving up on finding it and timestamping it with the previous event's timestamp or the current Indexer time.

ON RARE OCCASION the indexer has the wrong time (see the first point about NTP) and so _indextime is wrong.

Many people get tired of playing whack-a-mole on this problem and elect to let the Indexer's timestamp such that _inedextime is always used for _time by using DATETIME_CONFIG = CURRENT OR to let the forwarder do it by using DATETIME_CONFIG = NONE:

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Configuretimestamprecognition

I personally would not take the last approach but work to fix it PROPERLY, but sometimes "close enough is good enough".