How can I know what is wrong when there is a big difference in _time and index time
173,518 events (2/20/13 5:27:50.000 PM to 1/1/18 12:00:00.000 AM) No Event Sampling Job Fast Mode
Events
Statistics (173,518)
Visualization
100 Per Page
Format
Preview
Prev12345678...Next
_time idxtime offset _raw
2015-12-17 07:37:56.000 2018-08-14 04:54:59 83884623 timelag=423 messageId=1450337876eb4ae5bdd1fc7383fe8685 topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=353 messageId=0a9ec5de23bb4f32860895ae5474ea3e topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=257 messageId=228fd880217142c6806367ea28264c24 topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5 com.apple.keystone.messaging.client.v2.impl.kafka.ReceivedMessagesProcessor - "Kafka consumer received message" timelag=162 messageId=5383df5980ba4f4882cd464c31ef64aa topicName=KistaTopicNC3 retryCount=0 [LogLevel=INFO] -- 2018/08/14 04:54:30 INFO Thread-5
This could be due to indexer lag or bad timestamping. To test this, you can use this query. If it's linear then you most likely have a lag issue, if its flat then you most likely have a timestamping issue
You could also add a by source
and specify a host if you wanted to test your lag theory. Typically all the sources will have lag if the UF isn't keeping up
index=... sourcetype=...
| eval indextime=(_indextime, "%s")
| eval time=(_time, "%s")
| eval diff=time-indextime
| timechart span=1m max(diff) AS diff
@rajneeshc1981
can you confirm that _time=2015-12-17 07:37:56.000
and _indextime=2018-08-14 04:54:59
and that there is a multi year difference between your timestamps?
can you post the config in your props.conf
file for this sourcetype
? it might also help to get a copy of the inputs.conf
config for this sourcetype as well.
can you post a sample of the original raw source data before being sent to splunk and a sample of the _raw
after being received by splunk?
Check the various queue sizes if there is any high spikes on the queue sizes.
index=_internal sourcetype=splunkd source=*metrics.log group=queue
| timechart avg(current_size) by name
You can add host=yourUFName to see queue sizes on UF and host=Indexer (add more OR condition for all indexers) to see queue sizes on Indexers. You may need to adjust queue sizes based on results from there.
https://answers.splunk.com/answers/38218/universal-forwarder-parsingqueue-kb-size.html
how is queue size related to _time
how the timestamp is being extracted (_time doesn't seem to match the one in the raw data).
how to find it ? ,I don't know how it was extracted ?.
read this answer in detail
https://answers.splunk.com/answers/678655/how-to-trigger-alerts-when-indextime-time-1.html#comment-6...
hope it helps