Why am I unable to search previously indexed data?

[cykuan]

Hi All,

My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31", it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01", it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.

Is there any permission issue during search? I only made changes to admin role to inherit can_delete.

[cykuan]

I understand, I only deleted source="log.2015-05-22", but other source likesource="log.2015-05-23"or source="log.2015-06-01" should not be deleted and able to display the event, am I right?

If I want to re-index back, what should I do? I have already tried to re-index the source="log.2015-05-22", but there is no event showing anymore for this source.

[woodcock]

If you edit the file and swap the first 2 lines (move the top line down 1 line), it should re-index the file. The rest of what you are saying makes no sense unless you accidentally deleted more than you think you did.

[cykuan]

I know it sound weird, but it actually happen to me. For example, I put in a new log file(/home/user/cdr/chat.log.2015-06-02), when I try to do a search source="/home/user/cdr/chat.cdr.2015-06-02", there is no result at all. Any comments?

[woodcock]

Do this search for "All Time" just to make sure the events are not timestamped "in the future" or something way off from what you expect:

... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source

[cykuan]

Hi Woodcock,

I have tried the command you provided, and it's able to show some of the index files. The result only show log.2015-05-22 until log.2015-05-31. Since my oldest log file is log.2015-05-22, hence the result display is correct. However, my latest indexed file should display log.2015-06-02, unfortunately, it doesn't show up.

[woodcock]

Did you run it for "All Time"? This is very important (otherwise "future" events will not be found).

[cykuan]

Hi Woodcock,

Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000). Is this the reason that caused the Splunk can't show the event of 2015-06-01 onward?

[cykuan]

Yes, after I run the command for "All time", the source display all the log which start from log.2015-05-22 until log.2015-06-02. Since the log file of 2015-06-02 has been indexed, why I can't see the statistic display on my report? My report only show the statistic start from 2015-05-22 until 2015-05-31 only.