Getting Data In

Why am I seeing so many corrupt buckets?

kutzi
Path Finder

While searching for the reason why our indexers are creating so many hot buckets, I executed:
| dbinspect index=* corruptonly=true and saw that we have many corrupted buckets (> 70).
Most of them are corrupted because of count mismatch tsidx=... source-metadata=..., but also some with Cannot get slices.dat count

For example:
_internal~171~B8CCBABE-56EF-4E03-9CC1-0E3F674AE341 count mismatch tsidx=2093832 source-metadata=2093800 1556018538 2093800 B8CCBABE-56EF-4E03-9CC1-0E3F674AE341 1 171 _internal 04/23/2019:11:22:29 /opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_171 402303625 232.7734375 14 11 uni-spl-shd-02.livec.sg-cloud.co.uk 1555672065 hot full

What could be the reasons that there are so many corrupted buckets? One of the few infos, I found about bucket corruption, said that it might be caused by an indexer process crash, but we didn't experience one.
Is there a way to get more info about the reason of the corruption.

0 Karma

dchoi_splunk
Splunk Employee
Splunk Employee

if you're seeing the corrupt only on hot buckets, you need to change the SPL to retrieve.

|dbinspect index=* corruptonly=true | search state!=hot

Or

|dbinspect index=_internal corruptonly=true | search NOT state=hot

The reason we exclude hot buckets is because  the status of those hot buckets are in transient and they are still being updated(written).

RishiMandal
Explorer

Do you have any encryption agent installed on the indexers as part of any security standard to encrypt the sensitive data being indexed?

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...

Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...

After a whole week of being on call, you fell asleep on your keyboard, and you hit a sequence of buttons that ...