Deployment Architecture

How to configure auto loadbalancing on multiple indexes?

Jeremiah
Motivator

How should I allocate space for indexes among indexing nodes? For example, lets say I have 2 groups of servers that will write to two separate indexes, indexa and indexb. I know each will log 50 GB/day, I want to keep data from both indexes for 30 days. This means I will need approximately (50GB/day*30days*.5 compression) 750 GB of total space for each index. If I have two indexing nodes, should I split the storage up evenly by creating indexa and indexb on both servers, and set a max size of 375 GB on each (750GB/2)? Are there any caveats as we add more indexes or more search nodes? Do we need to over allocate space to account for potential unevenness in the load balancing (ie, server1 receives more traffic than server2)?

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Yes, you should split it evenly between both, assuming you're using Splunk forwarders to load-balance between the two. You should allocated more storage for unevenness, but this is hard to say. For example, a server going down, loading of historical or archived data, or writes of a large file all at once (vs appending of a log file over time) will tend to cause unevenness. As the number of servers goes up, unevenness in volume across indexers tends to go down.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Yes, you should split it evenly between both, assuming you're using Splunk forwarders to load-balance between the two. You should allocated more storage for unevenness, but this is hard to say. For example, a server going down, loading of historical or archived data, or writes of a large file all at once (vs appending of a log file over time) will tend to cause unevenness. As the number of servers goes up, unevenness in volume across indexers tends to go down.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...