Problems with indexes, buckets and indexed data

[nsanchezfernand]

Hi all.

We have several apps and indexes in Splunk. Suddenly, this morning, there were not data at any index (except indexes like _internal, _audit). By investigating we have seen that the majority of the indexes are disabled and because of this we have discovered that there was duplicated IDs of buckets (I don't know why). Searching in Splunk answers I have found how to fix the issue with duplicated bucket IDs (link text and I have follow these steps, hoping find data in the indexes after restart Splunk, but indexes does not have any data :(.

In the indexes view, there is some indexes that I can not enable (I don't know why), and the indexes that I can enable shows 0 Current Size. We have review the directory where the buckets are and there is data inside, because of this I don't know why searchs don't return any data.

We have also tried to upload new data to one index after enable it, and it seems to do it right in the "Add data" view, but when we finish the upload and select "Start search" it does not return any data. We have tried to create a new index and upload data into it, and it does not works. The index is created and it seems to upload data in it but when we search it does not returns any data, because the index is created disabled, how is it possible?

We have a big problem because no application is working 😞 I have been all day trying to get a solution, but really I don't know what is happening!!!

Thanks, thanks and thanks again.

[nsanchezfernand]

Hi all.

The issue has been resolved. After much investigation we have discovered that our admin role had been modified, and the permissions for all non internal indexes had been removed. Because of this we only had permissions (and they were showed like enabled) to read all internal indexes.

Finally, there were duplicated IDs of buckets but the problem was of permissions (I didn't know that if you don't have permissions for one index this appears in Settings>Indexes like unable...).

Thanks and regards.

[skoelpin]

Woohoo!! This is a great outcome to a scary problem! Good work @nsanchezfernandez!

You should take this moment get added to the entitlement now so you can get support in the future

[skoelpin]

It sounds like you have bucket collisions due to conflicting bucket ID's. Did you make a recent change that would cause this? Are you on a standalone box or are you working in a distributed environment?

If the buckets are there then the data is still there, but your indexes will be disabled until the bucket collisions are fixed.

This sounds like a P1, Splunk down issue. You should file a support ticket immediately

[nsanchezfernand]

Hello, Skoelpin.

Thank you for your response. I'm working on a standalone. I reviewed if there were duplicated IDs of buckets, and they were. I fixed by stoping Splunk, renaming the buckets with duplicated IDs and starting again Splunk, and now there are not duplicated IDs, but it still does not work. The indexes still appears like empty (0 current size) and disable (if I enable them and upload data the data is not available). I tried yesterday also to execute the command splunk fsck repair, but it didn't neither work.

The problem to open a support ticket to Splunk is that we are not proprietaries directly of the license, it is provided by other department of our company.

I'm really afraid, all of our entourage is unavailable :(.

[skoelpin]

I'm pretty sure you can contact Splunk and get added to the entitlement, or have them open the ticket and you can take ownership of the ticket.

Did you change the epoch timestamp on the bucket or just the bucket_id? If you changed the timestamp on the bucket, then it may be difficult to recover the data