Deployment client (occasionaly) fails to restart after removing app via editing the serverclass.conf

[gregbujak]

All Servers are - windows 2008 R2 on virtualized hardware.
All Splunk components running - Splunk: 5.0.3

It appears that when you remove a well defined App (contains both the local.meta and app.conf), there is a chance that the deployment client will not restart.

Steps:

• Modify the serverclass.conf on the deployment server so that the app in question will no longer be pushed to a forwarder
• reload the deployment server - bin>splunk reload deploy-server

The app is always deleted on the forwarder, but the forwarder sometimes doesn't restart.

It appears that there was an issue that if you don't have the app.conf or the local.meta files, your deployment client may not restart post update of the app. Is this a special case? Has anyone seen this behavior before?

---

In these logs, I am removing the win_infra_fwdr app by changing the serverclass.conf.

06-24-2013 11:29:46.169 -0400 WARN DeployedApplication - Uninstalling application: win_infra_fwdr
06-24-2013 11:29:46.169 -0400 WARN DeployedApplication - Removing app at location: C:\Program Files\SplunkUniversalForwarder\etc\apps\win_infra_fwdr
06-24-2013 11:29:46.174 -0400 WARN DeployedApplication - setting restart flag in context
06-24-2013 11:29:46.229 -0400 ERROR ConfObjectManagerDB - Cannot initialize: C:\Program Files\SplunkUniversalForwarder\etc\apps\win_infra_fwdr\metadata\local.meta: The system cannot find the path specified.
06-24-2013 11:29:46.245 -0400 WARN ConfPathMapper - Failed to open: C:\Program Files\SplunkUniversalForwarder\etc\apps\win_infra_fwdr\local\app.conf: The system cannot find the path specified.
06-24-2013 11:29:46.245 -0400 WARN DeploymentClient - Restarting Splunkd...
06-24-2013 11:29:46.937 -0400 INFO loader - Shutdown HTTPDispatchThread
06-24-2013 11:29:46.937 -0400 INFO ShutdownHandler - Shutting down splunkd


[LewisWheeler]

Anyone seen this past 5.x of UF - I am on 6.2.4 and have seen this across 8 servers all running Windows 2012 DC?

[bruceclarke]

@LewisWheeler, have you gotten anymore information on this? I'm seeing the same issue with 6.2.x.

[LewisWheeler]

No, im working on a new instance of Splunk running 6.3.x - just deployed 60+ forwarders and not come across this problem (although none are W2k12).

Whenever I contact support about tickets they generally just tell me to 'try a more later version' - not very helpful.

[davidpaper]

Ran into similar problems with 5.0.2. Backing the Deployment Server down to 5.0.1 solved them completely. No experience with 5.0.3 - 5.0.6 to know if the problems were completely fixed within the 5.x train.

[dstaulcu]

that behavior (timeout on restart) is a known issue throughout a few versions of Splunk v5. Issue appears to be gone in Splunk v6.

More details:
First listed as known issue in 5.0.1
Listed as resolved issue in 5.0.6
Issue id: SPL-61193 - Deployment server initiating a restart after application deployment, and
Windows based UF takes longer than Windows allows (30 seconds).