How to set a deployment server between Forwarders and Clustered environment

[himaniarora20]

Hi Team,

We have a search head cluster and indexer cluster in our current Splunk environment.  The data to the indexer earlier was provided by multiple forwarders which had the endpoint for the Indexer. Now, since it is a multi-indexer architecture, we need a common point for the forwarder to point the data

Please provide suggestions on how to set up the forwarders -> Deployment Server ->Cluster master architecture. I came across this one. But confused with the meaning of deployment client 

https://community.splunk.com/t5/Deployment-Architecture/How-to-set-up-new-deployment-server-in-a-clu...

Thanks in advance!

[himaniarora20]

Thanks for your reply. But I have a few questions since I am new to this.

1. In which server should I add the custom Add on (Forwarder or DS?) We have hundreds of forwarders pointing to the indexer right now. Do we need to change all of them?

2. And since you are saying I shall remove the already existing files in $SPLUNK_HOME/etc/system/local folder, what shall be the contents of the newly added custom add on files?

3. Also, the indexer discovery feature needs to be installed in the DS right? 

[gcusello]

Hi @himaniarora20,

the best approach is to create a custom add-on (called e.g. TA_Forwarders) containing at least three files:

• app.conf: containing the name and description of the add-on;
• deploymentclient.conf: containing the address of the Deployment Server;
• outputs.conf: containing the address of the Indexers.

then you should remove the same conf files from $SPLUNK_HOME/etc/system/local folder in each server.

In this way you can dinamically manage the indexers and Deployment Server addressing from the DS.

For indexers, you could also use the Indexer Discovery feature (https://docs.splunk.com/Documentation/Splunk/9.2.0/Indexer/indexerdiscovery) pointing to the Cluster Manager instead to the Indexers.

Indexers Cluster must be managed by the Cluster Manager not by the DS.

Search Head Cluster must be managed by the SHC Deployer not by the DS.

Ciao.

Giuseppe

[himaniarora20]

Thanks for your reply. But I have a few questions since I am new to this.

1. In which server should I add the custom Add on (Forwarder or DS?) We have hundreds of forwarders pointing to the indexer right now. Do we need to change all of them?

2. And since you are saying I shall remove the already existing files in $SPLUNK_HOME/etc/system/local folder, what shall be the contents of the newly added custom add on files?

3. Also, the indexer discovery feature needs to be installed in the DS right? 

[gcusello]

Hi @himaniarora20 ,

answering to your questions:

the custom Add-On must be located in every Forwarder.

If you have an already configured Deployment Server, you can load it in the DS and deploy it using the DS, but to be useful, you have also to remove the old conf files from the $SPLUNK_HOME/etc/system/local folder.

Otherwise the old conf files will con tinue to have precedence on the new ones.

Indexer Discovery, as you can read in the url I shared, must be configured in the outputs.conf file that must be located in the TA_Forwarders Add-On.

So it doesn't must be installed on the DS, but deployed to all the Forwarders using the DS.

Before starting this job, I hint to follow a training for Splunk Admin or engage a Splunk Admin (better an Architect), to assess your infrastructure, don't start your job without an adequate preparation!

Ciao.

Giuseppe

[isoutamo]

Hi

it seems that you have some misunderstanding for Splunk deployment architecture. Here is one document which show supported and proposed architectures for Splunk https://www.splunk.com/en_us/pdfs/tech-brief/splunk-validated-architectures.pdf.

As you can see there is no DS between indexers and UFs (or source systems). DS is just management server which define all needed apps (read configurations) which are needed on UF side to collect wanted events/logs/files from source systems. Those are sending all events (preferred) directly to indexers. 

@gcusello already told you how this configuration have done on DS side and what you need to do on UFs to get the new configuration in use (remove those from .../system/local/).

1) yes you must configure all those ../system/local if there is those configuration added installation time or later on. If you already use separate app(s) to manage those then it's not needed. Just update those apps as needed and DS update those into UFs.

2) Not all, just those which control DS/DC connection and if there are some additional inputs.conf, props.conf etc which are used to collect application logs from that system.

3) It depends on your environment. If you have static indexers (no additions, changes, deletions) then you can also use those IP/(I prefer) names on outputs.conf. But if your environment is dynamical then definitely you should use that. This needs to install all your UFs (via your dedicate app which define general index / site configuration) and also to all your Splunk infra nodes except indexers itself.

r. Ismo