Parametrized universal forwarders in a distributed env.

[greg0ry]

Perhaps I'm missing out on something but I haven't found details in the Splunk documentation about how we can setup generic universal forwarders that know "magically" on what deployment client they're running on.

We have a mixed environment with different breed of application servers and we'd want to centrally distribute a universal forwarder configuration that upon deployment (or later) would be intelligent enough to send log data with the correct syntax to the indexer.

All I've found so far is examples where parameters are hard coded but nothing dynamic.

How can we achieve this, is there a way to parametrize the forwarders during deployment (i.e. Forwarder_X will run on a Tomcat host, Forwarder_Y will run on an Orion Server host)?

One idea I haven't tried is with defining different server classes. After a successful deployment of forwarders one changes the inputs.conf settings on the deployment server centrally. Then a reload of server class:

./splunk reload deploy-server -class [server-class-name]

...would trigger an update of all forwarders and with that, we'd end up with client specific forwarders.

[kristian_kolb]

Well, you should NOT change the DS's own inputs.conf, but rather create a few applications - each containing an inputs.conf file.

On the DS, create an 'app' under $SPLUNK_HOME/etc/deployment-apps/ for each type of server.
Such an app could contain an inputs.conf file specifying [monitor] stanzas relevant for that type of server.

Then you create/edit the serverclass.conf (typically under $SPLUNK_HOME/etc/system/local/) on the DS, defining which servers should have what app.

Then you make sure that the forwarders know whom to contact for configuration information, i.e. run ./splunk set deploy-poll <ip:port> on each forwarder. (replace ip:port with your DS ip and port (8089 by default)).

Then you can run ./splunk reload deploy-server on your DS.

If everything went well, your forwarders should contact the DS, download the new app, and start sending the logs.

For more info, see

http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Aboutdeploymentserver
and the pages following.

hope this helps,

Kristian

[jbsplunk]

You're headed in the right direction, server classes configured so that Forwarder_X gets a Forwarder_X and and Forwarder_Y gets a forwarder_Y app is the solution. This is the normative method for accomplishing this type of task.