I missed some of the events for my search query, when I try to evaluate the time diff between event time and index time, Index time is early than event time. How can I resolve the timing issue? Where should I navigate and check??
If _indextime is earlier than _time, then one of the following is true:
The timestamping entity (the process writing to the file on the forwarder) is writing the wrong time. This is usually because the host OS has the wrong time. This is usually because the admin of that device neglected to setup NTP.
The timestamping entity is using a proper TZ but is not writing this TZ in the timestamp. This can be fixed by using the TZ=feature and deploying this to the forwarder (if post 6.0) or the Indexers (if pre-6.0).
The timestamping entity is using the wrongTZ and is writing this TZ in the timestamp. In the short-term, this can be fixed using the TZ_ALIAS= feature, but the proper fix is to correct the timestamping entity's insanity.
You have not correctly told Splunk where to find the timestamp (DO NOT EVER allow it to automatically timestamp: tell it exactly where and in what format the timestamp is) so it is mistaking non-timestamp data as a timestamp, misinterpreting the timestamp, or giving up on finding it and timestamping it with the previous event's timestamp or the current Indexer time.
ON RARE OCCASION the indexer has the wrong time (see the first point about NTP) and so _indextime is wrong.
Many people get tired of playing whack-a-mole on this problem and elect to let the Indexer's timestamp such that _inedextime is always used for _time by using DATETIME_CONFIG = CURRENT OR to let the forwarder do it by using DATETIME_CONFIG = NONE: