Alerting

Best Practices for Building A Clustered Environment?

woodlandrelic
Path Finder

Hello All,

I have been tasked with building a clustered environment from scratch in PROD. This will be my first.  I have only practiced in a test environment and everything is usually good. But, I would like to know any DOs and DONTs if any, or tips to be more successful.

Secondly, Once am done and everything is running how do I connect the old environment to the new one and Transfer or copy rather the same alerts, reports, dashboards, and apps to the new site?

Thanks for your help in advance.

 

Labels (1)
Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @woodlandrelic,

about the installation, I'haven't anything to add respect the installation and configuration procedure that you followed in test environment, that I suppose is the one described in Splunk on-line documentation.

About Apps, you have to copy all the apps present in your Master Node ($SPLUNK_HOME/etc/master-apps) and copy them in the same location od the new Master Node and deploy them to search peers.

For Search Heads, you have at first to create in the new environment all the roles of the old one.

Then, you have to copy all the Apps (excluding the ones bundled in Splunk installation) from a Search Head (possibly the Captain) and copy them in the Deployer to deploy them to all the Search Heads.

When you'll finish, you'll have a copy of your environment in the new one and you can switch the data flow.

To switch the data flow, you have to modify in each client, using the Deployment Server, the indexers addressing to the new one.

I suppose (it's a best practice) that you have outputs.conf in a dedicated TA, so it's easy to change it.

If you haven't outputs.conf in a dedicated TA but it's in $SPLUNK_HOME/etc/system/local, this is the opportunity to change this approach following these steps:

  • create the new outputs.conf addressing the new Indexers Cluster,
  • create a new TA (called e.g. TA_Forwarders) containing: apps.conf, outputs.conf and deploymentclient.conf,
  • copy it in the Deployment Server (in $SPLUNK_HOME/etc/deployment-apps),
  • create a ServerClass in serverclass.conf to deploy the new TA to all the clients,
  • deploy the new TA to all the clients,
  • manually (eventually using a script) remove outputs.conf and deploymentclient.conf from $SPLUNK_HOME/etc/system/local of each client,
  • BEWARE: do this step always after the deployment of the new TA, otherwise you'll loose all configurations!
  • restart Splunk on each client.

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @woodlandrelic,

about the installation, I'haven't anything to add respect the installation and configuration procedure that you followed in test environment, that I suppose is the one described in Splunk on-line documentation.

About Apps, you have to copy all the apps present in your Master Node ($SPLUNK_HOME/etc/master-apps) and copy them in the same location od the new Master Node and deploy them to search peers.

For Search Heads, you have at first to create in the new environment all the roles of the old one.

Then, you have to copy all the Apps (excluding the ones bundled in Splunk installation) from a Search Head (possibly the Captain) and copy them in the Deployer to deploy them to all the Search Heads.

When you'll finish, you'll have a copy of your environment in the new one and you can switch the data flow.

To switch the data flow, you have to modify in each client, using the Deployment Server, the indexers addressing to the new one.

I suppose (it's a best practice) that you have outputs.conf in a dedicated TA, so it's easy to change it.

If you haven't outputs.conf in a dedicated TA but it's in $SPLUNK_HOME/etc/system/local, this is the opportunity to change this approach following these steps:

  • create the new outputs.conf addressing the new Indexers Cluster,
  • create a new TA (called e.g. TA_Forwarders) containing: apps.conf, outputs.conf and deploymentclient.conf,
  • copy it in the Deployment Server (in $SPLUNK_HOME/etc/deployment-apps),
  • create a ServerClass in serverclass.conf to deploy the new TA to all the clients,
  • deploy the new TA to all the clients,
  • manually (eventually using a script) remove outputs.conf and deploymentclient.conf from $SPLUNK_HOME/etc/system/local of each client,
  • BEWARE: do this step always after the deployment of the new TA, otherwise you'll loose all configurations!
  • restart Splunk on each client.

Ciao.

Giuseppe

woodlandrelic
Path Finder

Hi @gcusello 

Thank you so much for the detailed answer. It was such a great help.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...