Deployment Architecture

When is it best to configure indexing across indexes using volumes?

thomas_forbes
Communicator

My current Splunk architecture plan is to have a combo search head and deployment server along with 2 indexers clustered together. What would be the best way to configure index storage? 1) Per unit basis, 2) Separate buckets based on temperature (hot, cold, etc), or 3) Across indexes, using volumes?

Also, is there a matrix of some sort that can indicate best practices when it comes to the actual size of the buckets? Based on our current plans we would fall into the small enterprise category.

Thank you very much in advance.

0 Karma
1 Solution

muebel
SplunkTrust
SplunkTrust

Generally you'll want to keep the warm/cold volumes on the same filesystem so that the warm -> cold rolling is efficient. You will only need to have multiple filesystems if you want to break out the cold into a cheaper sort of storage.

I would stick with the defaults for buckets size, unless you have a particular index where you will be doing more than 10GB per day, and then there is the auto high volume setting that can be used. Check out indexes.conf for more details. The documentation has more to say on configuring the volume locations.

View solution in original post

muebel
SplunkTrust
SplunkTrust

Generally you'll want to keep the warm/cold volumes on the same filesystem so that the warm -> cold rolling is efficient. You will only need to have multiple filesystems if you want to break out the cold into a cheaper sort of storage.

I would stick with the defaults for buckets size, unless you have a particular index where you will be doing more than 10GB per day, and then there is the auto high volume setting that can be used. Check out indexes.conf for more details. The documentation has more to say on configuring the volume locations.

somesoni2
Revered Legend

You already have Indexers in cluster, so just ensure all your forwarder, do load-balancing during sending data to indexers, so data is uniformly spread across both the indexers, regardless of bucket size, number of indexes. For best performance, use timebased load balancing.

See more here http://docs.splunk.com/Documentation/Splunk/6.2.5/Forwarding/Setuploadbalancingd

thomas_forbes
Communicator

The clusters have not been set up yet. We are in the process of gathering resource requirements for our servers that will host 2 clustered indexers and 1 Search Head/Deployment Server. I am attempting to put together sizing requirements for memory and hard disk space and what is the best way to configure indexing.

Thank you,
Tom

0 Karma

somesoni2
Revered Legend

I would go through Splunk Installation manual (link below) for that.

http://docs.splunk.com/Documentation/Splunk/6.1.9/Installation/Beforeyouinstall

0 Karma

thomas_forbes
Communicator

That doc did not really provide the information needed.

Back to the original question: What would be the best way to configure index storage? 1) Per unit basis, 2) Separate buckets based on temperature (hot, cold, etc), or 3) Across indexes, using volumes?

0 Karma

somesoni2
Revered Legend

I guess this should shed some light. (read the first para)

http://docs.splunk.com/Documentation/Splunk/6.2.5/Indexer/Usemultiplepartitionsforindexdata

Its recommend to use a single high performance file system to hold your index data for the best experience, but again they are costly. And if you've clarity on how frequently the historical/older data will be searched, your can have host/warm buckets in local/faster disk and cold db on slower/shared drives. Hope this helps in some way.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...