2 clusters vs clustered and unclustered vs etc/system/local

[richkappler]

We are running a large multi-site clustered indexer environment which is maturing causing us to make some changes to our hot/warm/cold rollover scheme. The one issue we have is 2 small sites have a different hardware setup than the rest of the environment. Due to this, I can't use the same indexes.conf on these 2 smaller sites that I use in the rest of the indexers.

The question then is what is the best approach to handling this situation? As I see it, I have 3 choices:
1. Run 2 clusters, which would force me to add another clustermaster.
2. Run the 2 smaller sites unclustered. My gut tells me this would be undesirable, but I'd like something a little more concrete than my gut.
3. Put an indexes.conf in etc/system/local for the smaller sites to override the indexes.conf we have in our slave-apps dir for the clustered indexers.

I believe option 3 to be the best but wanted to reach out for some verification and potential alternative suggestions.

[micahkemp · Accepted solution]

I'm going to make some assumptions that aren't necessarily stated in the original question:

a) You want to make a change to a configuration in indexes.conf in system/local to surgically make one or more changes to the version pushed from the cluster master.
b) This specific change isn't one that will impact the indexer's ability to be a valid member of the cluster ( [volume:] is an example of a parameter that can be different across indexers, and should be fine to set in system/local).

You can make changes to a clustered indexer in system/local to effect a change specific to only that indexer. Best practice warns against making any changes in system/local, and you can indeed make that change in apps/local according to the Precedence for cluster peer nodes documentation. Note that slave-apps/default takes precedences over apps/default, so it would have to be in apps/local to override slave-apps/local.

So, if you need some indexers to use different volume settings than others, I think this would be a valid method of running disparate configurations on clustered indexers.

[nickhills]

I think you are right, in that you can override indexes locally - volume is a good example of where you might wish to do this, however whilst you can specify that a given index exists on different disks, you cant specify different sizes of indexes, which I think is what the question is getting at.

[micahkemp]

I agree the initial question was framed inquiring about site-specific retention, which I didn't answer. I made some potentially invalid assumptions about the "question behind the question."

[nickhills]

Can I just check - when you say multi site, you mean 'Splunk Multi-site Cluster', rather than 1 cluster, simply running across multiple sites 🙂
I am assuming the answer is yes..

Have you configured site replication to unburden the small sites with multiple copies of data from the larger sites - this is not an answer to your specific question, but it might buy you some time.

On a previous deployment, I discovered that whilst you can not set a site to have 0 copies of another sites data, you can 'imply' it:
Assuming 5 sites: 3 big ones (s1, s2, s3), and two little (s4, s5):

site_replication_factor = origin:2, site1:2, site2:2, site3:1 total:5

Will mean that your small sites wont get replicated copies of sites 1-3's data, but they will always hold two copies of anything indexed locally. I cant find anything to say this is an officially supported approach, but the documentation does say:

..., the replication factor ... is not a
requirement. An explicit site is a
site that the replication factor
explicitly specifies. A non-explicit
site is a site that the replication
factor does not explicitly specify.

Which I read as "meh..it should be ok"

edit: added link https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Multisitearchitecture

I'm not sure option 3 would work, and I very much doubt that it would be supported - A cluster works on the idea that an entire indexes data exists in full in at least 1 other location - If that location has differing bucket life-cycles, then your remote site is not upholding its end of the bargain.

For the pain and problems you could otherwise be facing, could I offer option 4, which would read 'add more disks' ?
I know that is something you have probably deliberately omitted, but I cant help but think it would be your path of least resistance (until you submit a PO, anyway)

[richkappler]

Add more disks is not an option.

[richkappler]

I forgot to put the specific change, need for change here: the 2 smaller sites in question have smaller hotwarm disks, so the change would be hotwarm to cold rollover based on size (maxVolumeDataSizeMB).