How to deploy app to (and from) a multi-tiered deployment server

[steven_swor]

I'm setting up a multi-tiered Splunk deployment, with a primary and a secondary deployment server (where the secondary is a client of the primary). I want to:

1. Deploy the Unix TA from the primary to the secondary, so that the add-on will run on the secondary.
2. Deploy the Unix TA from the secondary to all of its clients, so that the add-on will run on them.

On my primary deployment server, I have the following in serverclass.conf:

# Drop the Unix TA into the secondary's deployment-apps folder,
# so it can deploy the add-on to its clients.
[serverClass:secondaryDeploymentServersDeploymentApps]
targetRepositoryLocation = $SPLUNK_HOME/etc/deployment-apps
whitelist.0 = SecondaryDeploymentServer
stateOnClient = noop

[serverClass:secondaryDeploymentServersDeploymentApps:app:Splunk_TA_nix]

# Install the Unix TA onto the secondary, so that we can collect
# its host metrics.
[serverClass:secondaryDeploymentServers]
whitelist.0 = SecondaryDeploymentServer
stateOnClient = enabled

[serverClass:secondaryDeploymentServers:app:Splunk_TA_nix]

When I start up the primary deployment server and view the forwarder management page, I see this message:

The forwarder management interface does not support some settings in your serverclass.conf file. The interface is now read-only.

When I click the link on the word settings, I get taken to a search with a single event, whose message reads:

Attribute unsupported by UI: stanza=serverClass:secondaryDeploymentServersDeploymentApps property=stateOnClient reason='2+ distinct values at this level'

I figured this was allowable, per the Splunk documentation, but I guess not. So, what is the correct way of setting this up?

[woodcock]

There is a bug in serverclass.spec that causes Splunk Deployment Server to give the following FALSE error logs:

Checking conf files for problems...
                Invalid key in stanza [serverClass:site1_indexers] in /opt/splunk/etc/system/local/serverclass.conf, line XXX: targetRepositoryLocation  (value:  $SPLUNK_HOME/etc/master-apps).
                Invalid key in stanza [serverClass:site1_indexers] in /opt/splunk/etc/system/local/serverclass.conf, line XXX+1: targetRepositoryLocationPolicy  (value:  rejectAlways).

You can confirm that it actually works by running /opt/splunk/bin/splunk show config serverclass | grep targetRepository which will give you this (which clearly indicates functional configurations):

targetRepositoryLocation=$SPLUNK_HOME/etc/apps
targetRepositoryLocation=$SPLUNK_HOME/etc/master-apps
targetRepositoryLocationPolicy=rejectAlways

You can double-check that it should work by running /opt/splunk/bin/splunk btool serverclass list --debug| grep targetRepository which will give you this (which clearly indicates valid configurations):

/opt/splunk/etc/system/default/serverclass.conf targetRepositoryLocation = $SPLUNK_HOME/etc/apps
/opt/splunk/etc/system/local/serverclass.conf   targetRepositoryLocation = $SPLUNK_HOME/etc/master-apps
/opt/splunk/etc/system/local/serverclass.conf   targetRepositoryLocationPolicy = rejectAlways

You can also confirm by checking the target Cluster Master destination which will clearly show that it does indeed pull in CM apps to $SPLUNK_HOME/etc/apps/ as well as Indexer apps (via a different serverclass using the targetRepository* settings) to $SPLUNK_HOME/etc/master-apps/.

It is unfortunate the both the documentation and the spec files are in error. I am trying to open a documentation update request to get this mess cleared up.

NOTE: You may, or may not, desire the rejectAlways setting. Use it or don't; it has nothing to do with the viability of the other setting(s).

[tomasmoser]

I followed this thread a have to say it does work. I am able to use DS to deploy selectively apps to both MN and IDXC cluser that MN manages and also to DEP and SHC cluster that DAP manages. This is awesome!

[wrangler2x]

Take a look at this Splunk answer and scroll down to the one which references https://gist.github.com/sworisbreathing/d236895568eaf31da579

[tomasmoser]

I followed hints in above link and finaly make my deployment working. See my comment below.

[mikaelbje]

Seems related to http://answers.splunk.com/answers/10500/tiered-deployment-servers-is-it-possible.html

I'd also very much like to see this functionality implemented and working, especially being able to manage clients 1 tier down from a central deployment server.