Getting error on Splunk restarts in the GUI after upgrading from 5.0 to 6.0 a few weeks ago.

[amgoldschmidt]

I keep seeing this on Splunk restarts in the GUI

FAILED_LOAD_DEPLOYMENT_SERVER__TENANT_default

Anyone know what it is or why I am seeing it ?

I did upgrade from Splunk 5.0 to 6.0 a few weeks back, so might be related to that.

[bosburn_splunk]

I've found some issues around the Deployment Server and how it handles missing applications - which may be part of the problem here. I've opened bug SPL-85551 and will let you know what is found.

[Adrian]

I also upgrade and was presented with the same error. I believe it is an artifact left behind from previous installation. You have to comment out your entries in /opt/splunk/etc/system/local/serverclass.conf

The entry that was present in my serverclass.conf was the following:

[global]
# whitelist.0=* at the global level ensures that the machineTypesFilter attribute
# invoked later will apply.
whitelist.0=*

[serverClass:AppsByMachineType]
machineTypesFilter=linux-x86_64

[serverClass:AppsByMachineType:app:unix]
# Deploy this app only to unix boxes - 32/64 bit.
machineTypesFilter=linux-x86_64

After commenting it looked like this:

#[global]
# whitelist.0=* at the global level ensures that the machineTypesFilter attribute
# invoked later will apply.
#whitelist.0=*

#[serverClass:AppsByMachineType]
#machineTypesFilter=linux-x86_64

#[serverClass:AppsByMachineType:app:unix]
# Deploy this app only to unix boxes - 32/64 bit.
#machineTypesFilter=linux-x86_64

I then restarted /opt/splunk/bin/restart and the error message disappeared.

[marksnelling]

I'm also getting this problem since upgrading to Splunk 6.0. I have a generic windows serverClass and a serverClass for SQL server.
The SQL serverClass defines it's own repositoryLocation for the TA-SQLServer app, somehow my generic windows serverClass is also using this repositoryLocation to locate the SA-ModularInput-PowerShell app instead of the one defined in [global].

Here are the relevant sections of my serverclass.conf. I haven't found a solution to this yet.

[global]
repositoryLocation = $SPLUNK_HOME/etc/deployment-apps
targetRepositoryLocation = $SPLUNK_HOME/etc/apps
stateOnClient = enabled
restartSplunkd = true
whitelist.0 = *

[serverClass:Windows]
machineTypesFilter = windows-x64
whitelist.0 = *
[serverClass:Windows:app:Splunk_TA_windows]
[serverClass:Windows:app:SA-ModularInput-PowerShell]

[serverClass:MSSQL]
whitelist.0 = sql01
repositoryLocation = $SPLUNK_HOME/etc/apps/Splunk_for_SQLServer/appserver/addons
[serverClass:SQLServer:app:TA-SQLServer]

[nwieseler]

I had the same issue: updated from 5.0.4 to 6.0 and started seeing that error.

Going through the logs using Splunk on Splunk (S.O.S.) sorting by ERRORS I found this in the splunkd.log:

11/26/13
2:12:16.801 PM

11-26-2013 14:12:16.801 -0600 ERROR loader - Deployment Server is not available, because errors prevented its initialization: 'Gap in numbered regexes: expected attribute=attrib.3 not found (context: stanza='serverClass:class_01')'. You can try to reload Deployment Server after correcting the errors.

Using the config file viewer I used the chrome search function to find matching stuff (in my case "attrib") which turned up in the serverClass file. The issue was I had was I had skipped a number in the sequence (below, it goes 0 1 2 4 - should be 0 1 2 3). In the log you can see 'Gap in numbered regexes: expected attribute=attrib.3 not found (context: stanza='serverClass:class_01')'

snip

C:\Program Files\Splunk\etc\system\local\serverclass.conf attrib.0 = redacted00

C:\Program Files\Splunk\etc\system\local\serverclass.conf attrib.1 = redacted01

C:\Program Files\Splunk\etc\system\local\serverclass.conf attrib.2 = redacted02

C:\Program Files\Splunk\etc\system\local\serverclass.conf attrib.4 = redacted04

snip

Going back through my backups I noticed that it's been that way for a long time. Seems like something is checking this file for errors now (I can only speculate as I have no clue).

Hope this helps...

Nick

[ShaneNewman]

11-25-2013 17:32:14.981 -0600 ERROR Application - Application=SA-ModularInput-PowerShell cannot be loaded, as path=C:\Program Files\Splunk\etc\deployment-apps\SA-ModularInput-PowerShell does not exist.
11-25-2013 17:32:14.981 -0600 ERROR DSManager - Failed to reload serverclass=MSSQL: Application=SA-ModularInput-PowerShell cannot be loaded, as path=C:\Program Files\Splunk\etc\deployment-apps\SA-ModularInput-PowerShell does not exist.
11-25-2013 17:32:14.981 -0600 WARN TenantService - Did not reload DSManager: Application=SA-ModularInput-PowerShell cannot be loaded, as path=C:\Program Files\Splunk\etc\deployment-apps\SA-ModularInput-PowerShell does not exist.
11-25-2013 17:32:14.981 -0600 ERROR loader - Deployment Server is not available, because errors prevented its initialization: 'Application=SA-ModularInput-PowerShell cannot be loaded, as path=C:\Program Files\Splunk\etc\deployment-apps\SA-ModularInput-PowerShell does not exist.'. You can try to reload Deployment Server after correcting the errors.
11-25-2013 17:32:14.981 -0600 INFO loader - win-service: Windows service is now in running state.

[delink · Accepted solution]

Search for "ERROR DSManager" in your splunkd.log. You will likely find the specific reason that the deployment server is not able to start correctly. If you have trouble interpreting the logs, post them here so we can take a look.

[delink]

If you are not using the deployment server, you should clear out the etc/deployment-apps directory and make sure you remove any non-system-default serverclass.conf configuration file. This should effectively disable the deployment server in that instance of Splunk.

Be warned that you should also remove any deploymentclient.conf configuration file from forwarders or other Splunk instances that you are running so they no longer try to check in.

[amgoldschmidt]

thanks, my error is "ERROR DSManager - Failed to reload serverclass=Unix: Application=unix_fwd cannot be loaded, as path=/opt/splunk/etc/deployment-apps/unix_fwd does not exist.
"

We are not even using the Deployment server.

[delink]

Yes, SOS does make this a lot easier, but if that is not installed already, getting it installed might be more difficult just to solve this issue. Especially without a working deployment server! 🙂

[nwieseler]

+1 for searching the logs... Splunk on Splunk makes this much easier (in my opinion anyway).

My issue was resolved viewing the splunkd.log file.

[rgoody]

Not sure but I am getting the same error after upgrade to version 6 so hopefully someone knows.