How can I know what is wrong when there is a big difference in _time and index time?

[rajneeshc1981]

How can I know what is wrong when there is a big difference in _time and index time

173,518 events  (2/20/13 5:27:50.000 PM to 1/1/18 12:00:00.000 AM)  No Event Sampling   Job Fast Mode
Events
Statistics (173,518)
Visualization
100 Per Page
Format
Preview
Prev12345678...Next
_time   idxtime offset  _raw
2015-12-17 07:37:56.000 2018-08-14 04:54:59 83884623    timelag=423 messageId=1450337876eb4ae5bdd1fc7383fe8685 topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=353 messageId=0a9ec5de23bb4f32860895ae5474ea3e topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=257 messageId=228fd880217142c6806367ea28264c24 topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=162 messageId=5383df5980ba4f4882cd464c31ef64aa topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 

[skoelpin]

This could be due to indexer lag or bad timestamping. To test this, you can use this query. If it's linear then you most likely have a lag issue, if its flat then you most likely have a timestamping issue

You could also add a by source and specify a host if you wanted to test your lag theory. Typically all the sources will have lag if the UF isn't keeping up

index=... sourcetype=...
| eval indextime=(_indextime, "%s")
| eval time=(_time, "%s")
| eval diff=time-indextime
| timechart span=1m max(diff) AS diff

[marycordova]

@rajneeshc1981

can you confirm that _time=2015-12-17 07:37:56.000 and _indextime=2018-08-14 04:54:59 and that there is a multi year difference between your timestamps?

can you post the config in your props.conf file for this sourcetype? it might also help to get a copy of the inputs.conf config for this sourcetype as well.

can you post a sample of the original raw source data before being sent to splunk and a sample of the _raw after being received by splunk?

[muralikoppula]

Check the various queue sizes if there is any high spikes on the queue sizes.
index=_internal sourcetype=splunkd source=*metrics.log group=queue
| timechart avg(current_size) by name

You can add host=yourUFName to see queue sizes on UF and host=Indexer (add more OR condition for all indexers) to see queue sizes on Indexers. You may need to adjust queue sizes based on results from there.
https://answers.splunk.com/answers/38218/universal-forwarder-parsingqueue-kb-size.html

[rajneeshc1981]

how is queue size related to _time

[muralikoppula]

how the timestamp is being extracted (_time doesn't seem to match the one in the raw data).

[rajneeshc1981]

how to find it ? ,I don't know how it was extracted ?.

[adonio]

read this answer in detail
https://answers.splunk.com/answers/678655/how-to-trigger-alerts-when-indextime-time-1.html#comment-6...

hope it helps