Deployment Architecture

Too many buckets - can I somehow merge them?

tkiss
Path Finder

Hi all,

In a 4 indexer cluster, where there are 60 individual indexes I happen to have 40.000+ buckets now (data is back from years). I assume this could cause some performance issues. Can you confirm this?

If yes, is there a way to somehow optimize the count of buckets? Merge buckets in the same index, eg: 10 buckets will become 1.

Using Splunk 6.1.4.

Thanks,
Tamas

Tags (3)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

This should not cause performance problems. It's possible that if you have tens of thousands of buckets in an individual directory, some older filesystems (e.g., ext3) may start hitting performance limits, but you're right now at an average of only 40000/(4x6) = 167 per directory, or even less if they are spread to a cold directory. Newer filesystems (e.g., ext4, XFS, NTFS) also avoid this.

You can't easily merge buckets, but you should make sure going forward that your indexes are set to a maxDataSize (max bucket size) of at least auto_high_volume (10 GB), and not auto (750 MB) to make sure they are not unnecessarily small.

View solution in original post

peppi
Explorer
0 Karma

tkiss
Path Finder

Thanks to both of you, currently I'm not seeing any performance impact - I'm just trying to be proactive here. maxDataSize is indeed set to auto_high_volume, anyway thanks for the tip!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This should not cause performance problems. It's possible that if you have tens of thousands of buckets in an individual directory, some older filesystems (e.g., ext3) may start hitting performance limits, but you're right now at an average of only 40000/(4x6) = 167 per directory, or even less if they are spread to a cold directory. Newer filesystems (e.g., ext4, XFS, NTFS) also avoid this.

You can't easily merge buckets, but you should make sure going forward that your indexes are set to a maxDataSize (max bucket size) of at least auto_high_volume (10 GB), and not auto (750 MB) to make sure they are not unnecessarily small.

martin_mueller
SplunkTrust
SplunkTrust

1000 buckets per index doesn't sound horrible to me. Are you actually seeing performance issues or are you just worried about the numbers?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...