is it possible to build a cluster with the heterogeneous index

[danielwan]

I have a all-in-one Splunk box. As more team are pushing their logs into Splunk, the current Splunk box is going to run out of disk (I have configured the retention policy)

What I want to do is to scale out the current 10+ indexes to different hosts and finally build a heterogeneous index cluster (each box hosts 2-3 unique indexes), meanwhile, provide a single end point(Splunk Web UI) for all users to do the searching. Is it feasible with Splunk? (According to Splunk document, the current index cluster apparently does not support the heterogeneous index)

[esix_splunk · Accepted solution]

What you are calling heterogeneous is what we call distributed. To accomplish what you want to do, you would need to build distinct indexers, or clusters. Then you can have a SH or Group of SH (SHC) search all these clusters or individual indexers.

You can limit access via roles and permissions or search filters to indexes.

Cheers
Eric

[cpetterborg]

By separating the indexes to different indexers you might have gains in one area, but losses in other areas. Splunk's distributed architecture across multiple indexes is well set up for making your searches work well across an entire cluster. By separating the indexes as you propose, danielwan, you are going against best practices.

One example of a loss by doing it that way is the disk space problem you are experiencing now. You may be trying to balance the addition of new hardware into your (clusters of) indexers. If you just do the best practice of distributing across all your indexers, then your disk space addition problem is easily solved - just add another indexer. You could end up needing 2 or 3 or 4 indexers to accomplish the same thing in your proposed scenario.

Is there some particular reason that you want to build out your environment in that way? You don't give any reasons in your question for doing so, which I think needs to be addressed.

[danielwan]

My Splunk server is running on a VM with a 250G volume. Disk usage has been over 90% even after applying retention policy. And a few more teams are going to push their log to Splunk

It's not the option to extend the storage capability, e.g. mounting a new volume or increase volume size, at this stage, so I am thinking it may be a path if it's possible to spread out indexes to different hosts (Currently, each team has a dedicated index)

[esix_splunk]

If your usage is growing to that point, perhaps you should discuss with management about investing in real hardware and making it a production tool with storage and HA..