Deployment Architecture

is it possible to build a cluster with the heterogeneous index

danielwan
Explorer

I have a all-in-one Splunk box. As more team are pushing their logs into Splunk, the current Splunk box is going to run out of disk (I have configured the retention policy)

What I want to do is to scale out the current 10+ indexes to different hosts and finally build a heterogeneous index cluster (each box hosts 2-3 unique indexes), meanwhile, provide a single end point(Splunk Web UI) for all users to do the searching. Is it feasible with Splunk? (According to Splunk document, the current index cluster apparently does not support the heterogeneous index)

Tags (1)
0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

What you are calling heterogeneous is what we call distributed. To accomplish what you want to do, you would need to build distinct indexers, or clusters. Then you can have a SH or Group of SH (SHC) search all these clusters or individual indexers.

You can limit access via roles and permissions or search filters to indexes.

Cheers
Eric

View solution in original post

esix_splunk
Splunk Employee
Splunk Employee

What you are calling heterogeneous is what we call distributed. To accomplish what you want to do, you would need to build distinct indexers, or clusters. Then you can have a SH or Group of SH (SHC) search all these clusters or individual indexers.

You can limit access via roles and permissions or search filters to indexes.

Cheers
Eric

cpetterborg
SplunkTrust
SplunkTrust

By separating the indexes to different indexers you might have gains in one area, but losses in other areas. Splunk's distributed architecture across multiple indexes is well set up for making your searches work well across an entire cluster. By separating the indexes as you propose, danielwan, you are going against best practices.

One example of a loss by doing it that way is the disk space problem you are experiencing now. You may be trying to balance the addition of new hardware into your (clusters of) indexers. If you just do the best practice of distributing across all your indexers, then your disk space addition problem is easily solved - just add another indexer. You could end up needing 2 or 3 or 4 indexers to accomplish the same thing in your proposed scenario.

Is there some particular reason that you want to build out your environment in that way? You don't give any reasons in your question for doing so, which I think needs to be addressed.

0 Karma

danielwan
Explorer

My Splunk server is running on a VM with a 250G volume. Disk usage has been over 90% even after applying retention policy. And a few more teams are going to push their log to Splunk

It's not the option to extend the storage capability, e.g. mounting a new volume or increase volume size, at this stage, so I am thinking it may be a path if it's possible to spread out indexes to different hosts (Currently, each team has a dedicated index)

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

If your usage is growing to that point, perhaps you should discuss with management about investing in real hardware and making it a production tool with storage and HA..

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...