How to manage universal forwarder configuration files using deployment server

[jtroanlw]

I've been searching high and low, but have been unable to find a complete-enough answer to our need to manage the configurations of all our universal forwarders with only the Splunk deployment server.

I've used a couple of test systems to try getting this to work, but the deploymentclient.conf file that's pushed out from the DS isn't recognized by the UF and it loses contact with the DS. I have found a workaround, in putting a link in the etc/system/local directory that points to the .conf file under the apps directory tree. (I'm now assuming that I'll have to do this with each .conf file that's pushed down from the DS.)

While this could work for all the .conf files during the initial deployment of the environment, I'd like to find a cleaner solution. (I've thought about linking the apps directory into the etc/system/local directory as a possibility.)

Other thoughts and approaches would be greatly appreciated. Thanks.....

[lguinn2]

Gack!! You should not be putting links from etc/system/local!! Stop, quick, before things go horribly wrong...

As you probably know, the forwarders poll the deployment server and pull apps whenever they are updated. When pulled, the new or updated apps are added to the SPLUNK_HOME/etc/apps folder on the forwarder.

However, there are two more things that should be done, that you might not know:

1 - the app must be enabled

2 - the UF needs to be restarted

My guess is that you haven't set these two characteristics on the deployment server. If you are configuring the deployment server from the Forwarder Management UI, look at the Applications Tab and Edit every application to set these flags.

If you are configuring the deployment server manually, edit serverclass.conf on the deployment server and add these two attributes for each app.

restartSplunkd = true
stateOnClient = enabled

After a manual edit, force the deployment server to rescan serverclass.conf with this command:

SPLUNK_HOME/bin/splunk reload deploy-server

I strongly recommend that you set nothing on etc/system/local on the forwarder except for the forwarder name in inputs.conf and server.conf. All configurations should go in an app. Of course, you will have to set the initial deploymentclient.conf file on each forwarder so that it can contact the deployment server for the first time! After that, let each forwarder pull the rest of its apps/configuration.

If you do it this way, all of the settings in the apps will take effect on the forwarders.

[jtroanlw]

I'm still in pure testing mode and the system will be rebuilt from scratch before going into production, so no harm done if I do accidentally toast my forwarder.

I did have both flags checked in the GUI (so that's taken care of) and I was already doing "reload deploy-server" after making changes to ensure my deploymentclient.conf changes were getting picked up ASAP. (I was also manually restarting the forwarder's splunkd.) In every case, the forwarder completely ignored the deploymentclient.conf file sitting in etc/apps/ (and having none in etc/system/local). After I put in the link, it worked as it was supposed to.

I would like to avoid using file links and manage all the .conf files completely through DS apps. That approach isn't working so far and the links appear to do exactly what I needed -- have the forwarder use the .conf files that it pulls down.

Following up on the last paragraphs.... if I set an initial deploymentclient.conf in etc/system/local, how do I get rid of it after the first phone-home? (I want the flexibility to change DS's in the future, using the DS to push a new .conf when the forwarder's home DS changes.)

Thanks.....

[lguinn2]

reload deploy-server is run on the indexer; that's probably what you meant, but it wasn't clear to me.

I would not set the initial deploymentclient.conf in etc/system/local - I would make an app specifically for it, maybe called deployment_settings or base_fowarder_settings

If you have reloaded the deployment server and restarted the forwarder and it still isn't working - file a support ticket. Something is wrong here.

[jtroanlw]

Your assumption about running reload deploy-server on the indexer is correct.

As the forwarder stands now, the deploymentclient.conf is in etc/apps/ and there's none in etc/system/local ... splunkd is reporting that it's not a deployment client.

I do have a couple of other active .conf files in etc/system/local to control the rest of the functions and intend to move those to apps next. I moved the deploymentclient.conf into my first app since it wouldn't affect the functionality of the other testing I was doing.

[maciep]

with regards to the last paragraph, for both the outputs and deployment client settings, we have apps on the deployment server to configure those settings.

When we install a forwarder, we manually copy those apps down to the forwarder (under etc/apps). So when it starts for the first time it knows where to send logs and who the deployment server is. We then deploy those same apps back down to the forwarder from the deployment server.

So when all is said and done, nothing is or ever was in system/local and we manage those setting from the deployment server.