Replicating serverclass.conf and deployment apps.

[SridharS]

This morning the Forwarder clients are timing out sometimes when talking to searchhead (also the headnode) which means it’s too busy again. ServerA is running as a Splunk server, but is not doing anything at the moment. I need to replicate the serverclass.conf and deployment apps from searchhead onto ServerA and get it setup as a deployment server. I’d like to redirect about half the Windows clients at it. May I know how that can be done. I know its quite simple but still I am facing some issues with that. Also we do not have clustering in place.

[esix_splunk]

While working with a tiered DS is possible, there are a lot of potential downfalls and as best practices go, we don't recommend this.

Case where localised or tiered deployments are required, what we recommend is using a global DS that is the initial point of contact for all Splunk instances. And then based on IP / Hostname whitelists, distribute a new Deployment App that tells the Splunk instance where to go for its real DS.

E.g., global-splunk-ds.mycompany.com, east-splunk-ds.mycompany.com, west-splunk-ds.mycompany.com.

All clients connect to the global-splunk-ds first, and based on the IP or Hostname, they will download a new deployment app which then redirects them to the EAST-splunk-ds or WEST-splunk-ds as defined by operational requirements.

This can be applied to Machinetypes also, any of the configuration options that the serverclass.conf recognizes per server class.

What this implies is that the GLOBAL-splunk-ds will have a minimal set of apps and that your localized / tiered DS's will contain all the apps for the servers connecting to them. Here, we most commonly would implement rsync to between the EAST / WEST DS to make sure all non-localized apps are in sync.

For rsync, Google has quite a bit already available for your searching pleasure.

[FritzWittwer_ol]

There have been some issues with tiered deployment servers in the past, see http://answers.splunk.com/answers/10500/tiered-deployment-servers-is-it-possible.html. I don't know if this has been fixed in between, but we are now using simple rsync to distribute our apps and serverclasses over the various deployment Servers we Need.

[SridharS]

Thanks! And may I know the concept of rsync and do you have a clustering environment enabled?

[FritzWittwer_ol]

• we use index clustering and are going for a search head cluster
• our master deployment server is a separate server which doubles as license server
• we have currently six additional deployment servers in different networks, they also act as Splunk proxy, running a Splunk enterprise but forwarding all events they receiver from the connected splunkforwarders to our index cluster
• we distribute all apps and the main serverclasses.conf via rsync over ssh from the master deployment server to the slave deployment servers
• each deployment server has a small specific serverclasses.conf which is loaded with the main serverclasses.conf, it contains one app specific for this deployment server. This app defines the outputs, telling the splunkforwarder to send all its output to the deployment server which is working as a proxy (or the index cluster if it is the master deployment server)
• we have a script which reloads the serverclass synces the files and reloads the serverclass.conf on each deployment server
• each connected splunkforwarder is configured to connect to exactly one of the deployment servers, lists of deplyoment servers are not supported in deploymentclient.conf. This file is written from our custom post install script

rsyc is an Unix program which synchronizes directories to remote systems, see https://en.wikipedia.org/wiki/Rsync

[SridharS]

That was very helpful thank you so much and may I know if making this change this will not affect the the apps in original search head ?repositoryLocation = $SPLUNK_HOME/etc/deployment-apps

And also may I know where to make the config changes to set serverA as a client of searchhead. Thanks in advance !

[diogofgm]

The change you're making on serverA deploymentclient.conf it will only affect serverA. The repositoryLocation its just says when serverA receives apps from a deployment server put them in this folder ($SPLUNK_HOME/etc/deployment-apps) rather than the default one ($SPLUNK_HOME/etc/apps).

For the changes needed to make serverA a client of searchhead i edited my answer. Also check the link i provided as it has a lot of information about deployment server configs.

[SridharS]

I tried that. I copy and pasted the serverclass.conf file to the ServerA. ran this command * ./splunk set deploy-poll [searchhead_ip]:8089* but it did not create a deploymentclient file. hence i created this file by my own

[deployment-client]
phoneHomeIntervalInSecs = 600

repositoryLocation = $SPLUNK_HOME/etc/deployment-apps
reloadDSOnAppInstall = true

[target-broker:deploymentServer]
targetUri = search_head:8089
phoneHomeIntervalInSecs = 600

But it didn't work. Now I have setup in ServerA as same as searchhead. But I am not able to see the apps in the ServerA splunk web. How to sort this out.

[diogofgm]

You can set that like you would for a tiered deployment server.
For the apps you can set serverA as a client of the searchhead to receive all the apps. On the ServerA deploymentclient.conf you put:

repositoryLocation = $SPLUNK_HOME/etc/deployment-apps

This will force the deployed apps to be place on ServerA deployment-apps rather then apps folder. Then you set serverA as a client of searchhead like you would for any other client. On serverA CLI run:

./splunk set deploy-poll [searchhead_ip]:8089

As for the other clients do you have a app for deployment configs? If so its just a mater of changing that. If not i'm afraid you'll need to change that on every client.

More on this:
http://wiki.splunk.com/Deploy:DeploymentServer (check last example)