Deploying and Managing 50+ Splunk forwarders

[AKG1_old1]

Hello,

I am looking for best approch for installing  and managing forwarders. we have everything on Linux. After reading some docs and posts I think deployment server is mostly used for managing forwarder.

Current Approch:
we use to have around 50 forwarder on different server at a time. when we have  to monitor new machine. we use bash script which

1. copy the forwarder template to remote machine
2. update inputs.conf in forwarder with host and monitoring path.
3. start the forwarder.

whenever we have to change something like adding new file to monitor, chaning path etc we use to manually update in forwarder. we dont keep same forwarder up its like once campaign over delete then and keep adding wheneven needed.

 I have tried using Deployment server and I am able to update static file on already installed forwarder.

Queries: 1. If we update inputs.conf in forwarder using deployment server. do I need to manually restart forwarder ?

Queries 2 : is it possible to change inputs.conf dynamically accoring to machine where its deployed. Aslo restart forwarder after change ?
example :
everytime when we deploy we have to change path (/net/hp707srv/hp707srv1/apps/QCST_MIC_v3.1.46_MASTER) and host (HOST123)

[monitor:///net/hp707srv/hp707srv1/apps/QCST_MIC_v3.1.46_MASTER/logs/mxlivebook/.../service.log]
disabled = false
host = HOST123
index = live
sourcetype = mx_java_event

3. Is it possible to install forwarder  using deployment server ?

Basially, I am looking to handle everything regarding forwarder from one certral location.

Thank you in advance for suggestions.

[ekost · Accepted solution]

Good day.

Yes, the Deployment Server (DS) is for forwarder configuration management. It's not designed for installing the forwarder on hosts. 

Queries: 1. If we update inputs.conf in forwarder using deployment server. do I need to manually restart forwarder ?

No. There's a switch you can set on the DS for each app or add-on it manages to initiate a service restart on a forwarder that pulls the app/add-on. See serverclass.conf for 

restartSplunkd

Queries 2 : is it possible to change inputs.conf dynamically accoring to machine where its deployed. Also restart forwarder after change ?

Two things:

# You can use the DS UI to set various regex-based filters under a [serverclass] and assign each serverclass different apps. This allows you to have multiple very simple and very similar apps with inputs.conf files with unique paths. So when a forwarder host checks in to the DS, based upon those filters it can receive the app with an input that's slightly different.

# The monitor stanza in the inputs.conf offers multiple wildcard and/or regex options. Depending upon the complexity of the path change for each host, this might be very easy to define using wildcards and regex, or it might not. I suggest creating another Answers post with details and scenarios specifically on that issue if you're looking for detailed guidance.

And yes, the DS can ask the forwarder to restart itself after an app is deployed.

3. Is it possible to install forwarder using deployment server?

The DS isn't designed to perform installations. If you modify the script you already use to deploy and install the forwarder package and add a deploymentclient.conf file, any newly installed forwarders will check in to the DS and, based upon the machine and host filters you define, pull down the configurations that match the host and optionally restart the forwarder services. 

Good luck!