topic Re: Why am I unable to search previously indexed data? in Splunk Search

Why am I unable to search previously indexed data?

cykuan — Tue, 02 Jun 2015 14:08:15 GMT

Hi All,

My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31", it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01", it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.

Is there any permission issue during search? I only made changes to admin role to inherit can_delete.

Re: Why am I unable to search previously indexed data?

woodcock — Tue, 02 Jun 2015 17:19:06 GMT

How big is your index space (in indexes.conf)? Splunk will automatically expire data (FIFO) and if you have a very small amount of space for you index, it could have already aged out. Use this search to check your retentions:

index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex "/splunk(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 02:09:33 GMT

Hi Woodcock,

Below is my Splunk indexes.conf, and when I run your search command that you provided, the retention days is around 34.857905.

maxTotalDataSizeMB = 500000

Re: Why am I unable to search previously indexed data?

woodcock — Wed, 03 Jun 2015 03:31:39 GMT

By any chance, are you using | delete somewhere to delete your events?

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 05:33:01 GMT

Hi Woodcock,

Yes, I did a |delete before to delete one event.

I did a source="log.2015-05-22" | delete, will this affect my whole result? I thought I only delete this event, and the other event should be able to display.

Re: Why am I unable to search previously indexed data?

woodcock — Wed, 03 Jun 2015 13:06:36 GMT

Whatere events were returned by the same command without the | delete command will all be deleted when yoi tack that on so all events from that source are now gone. So that explains everything, right?

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 13:57:14 GMT

I understand, I only deleted source="log.2015-05-22", but other source likesource="log.2015-05-23"or source="log.2015-06-01" should not be deleted and able to display the event, am I right?

If I want to re-index back, what should I do? I have already tried to re-index the source="log.2015-05-22", but there is no event showing anymore for this source.

Re: Why am I unable to search previously indexed data?

woodcock — Wed, 03 Jun 2015 14:19:01 GMT

If you edit the file and swap the first 2 lines (move the top line down 1 line), it should re-index the file. The rest of what you are saying makes no sense unless you accidentally deleted more than you think you did.

Re: Why am I unable to search previously indexed data?

jacobwilkins — Wed, 03 Jun 2015 14:26:51 GMT

Just to be safe, try this:

earliest=-90d@d source="log.2015-05-31"

If that works, your issue is just the time range of your search.

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 14:43:20 GMT

I know it sound weird, but it actually happen to me. For example, I put in a new log file(/home/user/cdr/chat.log.2015-06-02), when I try to do a search source="/home/user/cdr/chat.cdr.2015-06-02", there is no result at all. Any comments?

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 14:43:53 GMT

I have tried this, but still the same, no result at all.

Re: Why am I unable to search previously indexed data?

woodcock — Wed, 03 Jun 2015 15:33:44 GMT

Do this search for "All Time" just to make sure the events are not timestamped "in the future" or something way off from what you expect:

... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source

Re: Why am I unable to search previously indexed data?

cykuan — Wed, 03 Jun 2015 16:43:46 GMT

Hi Woodcock,

I have tried the command you provided, and it's able to show some of the index files. The result only show log.2015-05-22 until log.2015-05-31. Since my oldest log file is log.2015-05-22, hence the result display is correct. However, my latest indexed file should display log.2015-06-02, unfortunately, it doesn't show up.

Re: Why am I unable to search previously indexed data?

woodcock — Wed, 03 Jun 2015 16:56:37 GMT

Did you run it for "All Time"? This is very important (otherwise "future" events will not be found).

Re: Why am I unable to search previously indexed data?

cykuan — Thu, 04 Jun 2015 02:16:47 GMT

Yes, after I run the command for "All time", the source display all the log which start from log.2015-05-22 until log.2015-06-02. Since the log file of 2015-06-02 has been indexed, why I can't see the statistic display on my report? My report only show the statistic start from 2015-05-22 until 2015-05-31 only.

Re: Why am I unable to search previously indexed data?

woodcock — Thu, 04 Jun 2015 03:11:19 GMT

OK, based on what you just wrote, the problem is now clear: you have a tomestamp peoblem that is putting nowish/newish events into the future. Such events are only searchable with All time. The problem is probably timezone related. This search will help you focus in on and track the problem. You need to get the numbers in the range of 100-1000 (typical):

... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source

Re: Why am I unable to search previously indexed data?

cykuan — Thu, 04 Jun 2015 08:36:20 GMT

Hi Woodcock,

Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000). Is this the reason that caused the Splunk can't show the event of 2015-06-01 onward?

Re: Why am I unable to search previously indexed data?

cykuan — Thu, 04 Jun 2015 10:31:12 GMT

Hi Woodcock,

On my search command, I always search as a "All Time" period. For example, my log file has a keyword "Call_successful", then I put this keyword on the search bar and click "All Time". My first page of the first event is showing the log event of 2015-05-31 but the last page is showing log event of 2015-06-02. I noticed on the Time column the time format is different, for log event of 2015-05-31 the time format is 5/22/15 12:46:04.000 PM but for log event of 2015-06-02 is 2/6/15 11:49:23.000 PM, I believe this caused the time order mismatch, is there anyway to fix it?

Re: Why am I unable to search previously indexed data?

woodcock — Thu, 04 Jun 2015 13:48:27 GMT

Yes, you need to explicitly tell Splunk where the correct timestamp is using the TIME_PREFIX directive. You can create a RegEx for it that is flexible (e.g. use the 5th field if it exists but if not, use the 4th field).

Re: Why am I unable to search previously indexed data?

cykuan — Thu, 04 Jun 2015 16:19:51 GMT

Hi Woodcock,

In this case if I want to change the _time format to 5/22/15, as I highlighted as red color, please refer the screenshot (http://imgur.com/cGtFMdu). Some of my event has a different _time format, that's is the reason why it can't show the 1st Jun 2015 event, it is because the format is different from May, when you look at the screenshot then you will have a clear picture for my issue. I want to have _time with a same format, so it can display correct date when I do a timechart span=1d command.