Last indexed event is in the future

[r_devos]

I've noticed that the last indexed event in my Splunk instance is set to 19 Jan 2038.
I have tried to find this event with the following search query:
host=* | head 5. But this search just returns the latest events of today, not the ones of the year 2038. Searching with a custom time didn't work either: Splunk incorrectly states that the end date is before the start date when you try to search in the year 2038. Searching with the keywords latest and earliest also didn't find this event.

So my question is: how do I find this event and how do I get rid of it? (I would include screen shots but it doesn't let me because my karma is too low)

[andreasz]

I think your metadata are corrupt.

You can see the corrupted metadata with the following statement:

| metadata type=sourcetypes | stats max(lastTime) as max | eval max=strftime(max,"%c")

max

Tue Jan 19 04:14:07 2038

Here you can find a description how to check and repair Metadata:
http://wiki.splunk.com/Check_and_Repair_Metadata

[andreasz]

sorry, I forgot to mention, that you have to set the timeline to: Real time -> All Time (real time)
Than you should see the result

[r_devos]

Thanks for your answer. I tried the search you mentioned but it returned the latest event of today, not the one of 2038. (I also tried setting a custom time, but this didn't help either).
To make matters worse, after trying to repair the metadata, Splunkd refuses to start again (Splunkweb starts just fine).
Fortunately this was just a test set-up, I'll start again from scratch and I just hope this problem doesn't return.

[r_devos]

Thank you for your answers so far, unfortunately I still have not found the event from January 2038.
I've tried with 'latest' but it didn't show any events for the year 2038.
When I tried with earliest=+1d I got the following error message "Failed to start search on peer '...'."
Splunk is running on Windows Server 2008 R2 (x64), so the UNIX time "ending" on 19 Jan 2038 is probably not the root cause. I have however one Linux server sending event data to Splunk, so maybe that Linux server sent an event with a corrupt time stamp...

[kristian_kolb]

host=* earliest=+1d

will let you find events from the future, so-to-speak.

[rgcurry]

The problem with year 2038 may be due to UNIX Epoch time if your Splunk instance is running on a 32-bit UNIX system. Note the following:

"At 03:14:08 UTC on 19 January 2038, 32-bit versions of the Unix time stamp will cease to work, as it will overflow the largest value that can be held in a signed 32-bit number. ..."

Retrieved May 15, 2013 from http://en.wikipedia.org/wiki/Unix_time.

[sdaniels]

Sounds like a timestamping format issue. Can you find the event by doing this: 'latest=26y' or is that what you tried already?