Event Timestamp mismatch with Index Timestamp

[anandhalagaras1]

Last year 2019 we have deployed Splunk Cloud in our environment . Post which we have configured the logs into Splunk Cloud and the data seems to be searchable as per the retention policy post retention policy the data seems to be getting deleted from the system so this is how it should work.

So when i searched the data for few indexes i was shocked to see the data is available from 2014 in Splunk Cloud. We have deployed Splunk Cloud only last year then how come is it showing the event timestamp as 2014 date. And also when i search with index=* for 2014 year as a whole i can see more than 20000+ events and shocked.

When i checked the log files of those data from 2014 in few source files i can see that the log file doesn't have the date in it and only the time is present so if the date is not present in the log file it should take the system time but still how come it assign with the event timestamp as 2014.

And another set of logs i can able to see the log file with latest timestamp i.e with date and time in it but still the event time seems to be assigning with 2014 date and time. Its getting confused a lot why come it is assigning with 2014 date and time.

So I have ran the query adding the index time field into the query for the whole year of 2014 as mentioned below:

index=* host=* source=*
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")
| stats count by _time indextime host index source sourcetype

I can see the index time with latest timestamp whereas the eventtime seems to be with 2014 date and time. Its really confusing how come the recently indexed data (Based on index time) it is assigning with very old timestamp.

So kindly let us know how it works. Or what is the issue behind it. How the bucket concept work in Splunk Cloud?

Any one can help on this request please.

[kamlesh_vaghela]

@anandhalagarasan16021988

Yes,
It is possible to have _time from back days and _indextime is recent. This is due to the configuration done for TIMESTAMP extractions. Can you please check the extraction configuration (props.conf & transforms.conf for that sourcetype/source/ related events? Most of the cases, you can find time value in raw event which is used as TIMESTAMP during extraction. Just check your configuration you will get your answer.

[anandhalagaras1]

Can anyone help on the same.

[anandhalagaras1]

Can anyone able to help me on this.

[anandhalagaras1]

Can anyone help on my request.

[anandhalagaras1]

Can anyone help on the same.