Add capacity to indexer cluster

[splunkreal]

Hello guys,

we have 3 'hardware' indexers in a clustered environment (RAID), all physical disk slots are full , replication factor 3 and may be running out of space in a near future.

So is it possible to add new/higher storage indexers to this existing cluster in order to add capacity? Also is it possible to create new indexes ONLY on those NEW indexers and how?

Thanks.

[bjarnedein]

Hi Guys,

Above is a most interesting question, and I'd like to extend it even further...

Say you have an Index Cluster with 10+ servers already running each with 12 core CPU's, and we need more cores in the cluster to deal with the raising demand for ingesting even more events coming in.
Seen in the light of running it all virtual hosts (Linux) on VMWare, what will happen to the Index Cluster if we add another 5 Index Servers to the cluster - each with less core's (6 each)?

In other words: Even though it might not be the most optimal solution, will the Index Cluster still benefit from adding more servers with less cores each (compare to existing), or will that make it even worse.

PS. The reason for asking is that right now it's much faster to get new servers with 6 cores.

I'd be most happy to get some input on this subject, and in general hear a bit more about how "intelligent" the different Splunk instances are in dealing with divergence in capacity in clusters (Indexer and Search Heads).

Best Regards,

Bjarne Dein

[s2_splunk]

Your best bet is to add another cluster peer, that is very straightforward to do. Once you have done that, you can opt to rebalance your cluster to distribute existing data more evenly across all nodes.

Option 2, as @cusello pointed out, is to create a SAN/NAS-hosted mount point on each cluster member and reconfigure your index settings to roll old data off to SAN/NAS. A more complex undertaking and potentially more costly unless you already have shared storage in your environment.

To get you some breathing room, you can temporarily reduce your RF to 2 and remove excess buckets to prevent running out of disk space while you are adding more capacity. You can increase it again once you have the required storage online to meet your disk requirements, given RF/SF, retention and daily ingest.

Finally, no, you do not get to have certain indexers only host certain indices. All cluster peers are required to have the exact same configuration, pushed from the cluster master.

[gcusello]

Hi realsplunk,
the possibility to add storage to existing physical Indexers depends by the characteristics of these servers(if they can, it's possible!), Splunk uses available storage.

I don't think that it's possible to select which indexes put on some Indexers: Splunk cluster replicates indexes data between the clustered Indexers based on the Replication and Search Factors.

If you have this problem, you could use the Splunk's feature to select different storage for different types of data:

• Hot and Warm data on high performance disks,
• Cold data on slower disks.

So you can think to move cold data on slower disks (e.g. SAN or NAS) and use physical disks for hot and warm data, without change Indexers hardware configurations.

Bye.
Giuseppe

[gcusello]

If this answer satisfies your question, please accept or upvote it.
Bye.
Giuseppe