Why are historical events not showing up in searches?

[amoore12]

I needed to restart my Splunk instance on our heavy forwarder the other day. After restarting, I am unable to search on anything before the restart on our main indexer. I am not seeing anything in the logs to indicate an issue. If I browse out to the DB directories, I see the older data still there. The search summary shows the stats correctly. I have restarted the Splunk process several times in the past with no issues. Any ideas how to correct this issue? Any help would be great. Thanks.

[lukejadamec]

If you have made no changes to your index retention policy, then you should have no local/index.conf files with policy settings. If you do have local/index.conf files, they should not contain DEFAULT settings.
Here is some info on index.conf settings:
http://wiki.splunk.com/Deploy:BucketRotationAndRetention

[lukejadamec]

What are the permissions on the hot buckets?

[amoore12]

Now that I stopped Splunk and started it back up again. I can only search back to a few minutes ago. Very strange.

Yes permissions are the same

[amoore12]

I ran the splunk fsck --all. No results were returned.

[lukejadamec]

Those permissions seem a bit restrictive. Are they the same on the searchable and non-searchable buckets?

[amoore12]

db_1374688147_1374668673_3725_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9
Wednesday, July 24, 2013 1:49:07 PM to Wednesday, July 24, 2013 8:24:33 AM

db_1375807093_1375787134_3841_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9
Tuesday, August 06, 2013 12:38:13 PM to Tuesday, August 06, 2013 7:05:34 AM

*Data is contained in that file*
[root@feprsplunk01 db_1375807093_1375787134_3841_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9]# ls -l
total 622264
-rw------- 1 root root 451868154 Aug 6 09:34 1375807093-1375787134-2736230854333308492.tsidx

[lukejadamec]

You can also run a check on your buckets.

splunk stop
splunk fsck --all
splunk start

Here are the details on "splunk fsck"
http://wiki.splunk.com/Community:PostCrashFsckRepair

[lukejadamec]

You know, the buckets are named 'earliesttimestamp_latesttimestamp_uniquenumber', and the times are epoch times. Convert one or two of the non-searchable ones: http://www.epochconverter.com/

[amoore12]

Yes, if I go to say the windows DB directory via ssh, I can see all the new and old files.

Yes, I can search all indexes and get only the new events since restarting.

[lukejadamec]

Let me see if I have this straight.

1) You can see the searchable and non-searchable buckets in the Same index/database.
2) When you search that index/database you only see events since the restart?

[wagnerbianchi]

Maybe the buckets configuration. Could you check configurations?

[wagnerbianchi]

Check that in indexes.conf

[amoore12]

Is there a global setting or is that done per index? If global, what file am I looking for?

[amoore12]

Yes the unix servers are not going to the heavy forwarder. All events since the restart have been indexed. Yes, splunk has done something crazy. I was wondering if there is a corrupt DB file or something since splunk starts with newer events and goes backwards.

[lukejadamec]

You win the splunk did something weird award.

I'm gonna have to think about that one.

[amoore12]

It is indexing new events as it should from both unix and windows. What I lost were events prior to restarting the service. I can search on all events since the restart.

[lukejadamec]

So, you're not getting data from either unix (not passing through the heavy forwarder, or from windows?

[amoore12]

On the main indexer, I have syslog and Splunk forwarders from unix. On the heavy forwarder, it is just windows servers from Splunk forwarders that I am filtering out some events before forwarding on.

[lukejadamec]

What type of inputs?

[dhimanv]

What solution found in this case?

Was issue sorted out because I am also facing the same kind of issue.