topic Best Practices for Building A Clustered Environment? in Alerting

Best Practices for Building A Clustered Environment?

woodlandrelic — Tue, 13 Sep 2022 12:50:45 GMT

Hello All,

I have been tasked with building a clustered environment from scratch in PROD. This will be my first. I have only practiced in a test environment and everything is usually good. But, I would like to know any DOs and DONTs if any, or tips to be more successful.

Secondly, Once am done and everything is running how do I connect the old environment to the new one and Transfer or copy rather the same alerts, reports, dashboards, and apps to the new site?

Thanks for your help in advance.

Re: Building A Clustered Environment.

gcusello — Tue, 13 Sep 2022 06:45:45 GMT

Hi @woodlandrelic,

about the installation, I'haven't anything to add respect the installation and configuration procedure that you followed in test environment, that I suppose is the one described in Splunk on-line documentation.

About Apps, you have to copy all the apps present in your Master Node ($SPLUNK_HOME/etc/master-apps) and copy them in the same location od the new Master Node and deploy them to search peers.

For Search Heads, you have at first to create in the new environment all the roles of the old one.

Then, you have to copy all the Apps (excluding the ones bundled in Splunk installation) from a Search Head (possibly the Captain) and copy them in the Deployer to deploy them to all the Search Heads.

When you'll finish, you'll have a copy of your environment in the new one and you can switch the data flow.

To switch the data flow, you have to modify in each client, using the Deployment Server, the indexers addressing to the new one.

I suppose (it's a best practice) that you have outputs.conf in a dedicated TA, so it's easy to change it.

If you haven't outputs.conf in a dedicated TA but it's in $SPLUNK_HOME/etc/system/local, this is the opportunity to change this approach following these steps:

create the new outputs.conf addressing the new Indexers Cluster,
create a new TA (called e.g. TA_Forwarders) containing: apps.conf, outputs.conf and deploymentclient.conf,
copy it in the Deployment Server (in $SPLUNK_HOME/etc/deployment-apps),
create a ServerClass in serverclass.conf to deploy the new TA to all the clients,
deploy the new TA to all the clients,
manually (eventually using a script) remove outputs.conf and deploymentclient.conf from $SPLUNK_HOME/etc/system/local of each client,
BEWARE: do this step always after the deployment of the new TA, otherwise you'll loose all configurations!
restart Splunk on each client.

Ciao.

Giuseppe

Re: Building A Clustered Environment.

woodlandrelic — Wed, 14 Sep 2022 11:33:54 GMT

Hi @gcusello

Thank you so much for the detailed answer. It was such a great help.

Re: Best Practices for Building A Clustered Environment?

man03359 — Mon, 27 May 2024 10:18:01 GMT

Hi @woodlandrelic ,

Hope this message finds you well.

I have recently moved from out of a Splunk developer role to an admin role. I have to build a cluster environment out of scratch.

I have the basic understanding of a clustered environment but haven't setup yet.

Could you please guide me how can I start. Like what type of knowledge/ information gathering need to do with the client or customer before head. Also if there is any procedure/ order of components to follow.

It will be really helpful for me.

Thanks in advance 🙂