All Apps and Add-ons

Why does Splunk DB Connect fail to start after upgrading to 3.1.0?

peteror
Engager

I've just upgraded Splunk DB Connect to 3.1.0 from 3.0.6 and I get the "DBX Server is not available, please make sure it is started and listening on 1025 port or consult documentation for details" error. (I set the port to 1025 from the default as part of trying to troubleshoot, but it did not work with the original port 9998 either.

In splunkd.log I see this:

07-01-2017 14:29:54.225 +0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\windows_x86_64\bin\server.exe"" action=task_server_start_failed error=javax.servlet.ServletException: io.dropwizard.jersey.setup.JerseyServletContainer-4ee5d723@fb567993==io.dropwizard.jersey.setup.JerseyServletContainer,1,false stack=org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:661)\\org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)\\org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)\\org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)\\org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)\\org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\com.codahale.metrics.jetty9.InstrumentedHandler.doStart(InstrumentedHandler.java:103)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:252)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.server.Server.start(Server.java:411)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.Server.doStart(Server.java:378)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)\\io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:44)\\io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)\\io.dropwizard.cli.Cli.run(Cli.java:75)\\io.dropwizard.Application.run(Application.java:79)\\com.splunk.dbx.server.bootstrap.TaskServerStart.startTaskServer(TaskServerStart.java:97)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:59)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:108)\\

In splunk_app_dbconnect_dbx.log I get these

after installing 3.1.0:

2017-07-01T09:23:33+0400 [ERROR] [admin.py], line 421: Failed to fetch DMC settings to verify status
2017-07-01T09:23:33+0400 [ERROR] [admin.py], line 422: [HTTP 404] https://127.0.0.1:8089/services/dmc-conf/settings/settings; [{'code': None, 'type': 'ERROR', 'text': 'Not Found'}]
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 418, in _is_DMCDisabled
    dmc_settings = en.getEntity(DMC_SETTINGS['url'], 'settings')
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 249, in getEntity
    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 550, in simpleRequest
    raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body))
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/dmc-conf/settings/settings; [{'code': None, 'type': 'ERROR', 'text': 'Not Found'}]

after a restart:

2017-07-01T12:36:56+0400 [ERROR] [root.py], line 663: DJANGO: There was an error starting:
2017-07-01T12:36:56+0400 [ERROR] [root.py], line 664: The SECRET_KEY setting must not be empty.
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 661, in run
    configure_django(global_cfg)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 738, in configure_django
    if should_start_django() or FORCE_ENABLE_DJANGO:
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 771, in should_start_django
    return len(settings.DISCOVERED_APPS) > 0 and not isLite()
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 53, in __getattr__
    self._setup(name)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 152, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
ImproperlyConfigured: The SECRET_KEY setting must not be empty.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Bus STARTING
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Set handler for console events.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread '_TimeoutMonitor'.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Serving on 127.0.0.1:8065
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Bus STARTED
2017-07-01T12:36:57+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread 'Monitor'.
2017-07-01T12:36:57+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - - [01/Jul/2017:12:36:57.298 +0400] "HEAD /favicon.ico HTTP/1.1" 303 124 "" "Splunk/6.6.0 (Windows Server 8.1 Standard Edition; arch=x64)" - 59575f294c40e8e6a6d8 16ms
2017-07-01T12:37:07+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread 'Monitor'.
2017-07-01T12:37:07+0400 [ERROR] [config.py], line 138: [HTTP 401] Client is not authenticated
Traceback (most recent call last):
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\config.py", line 136, in getServerZoneInfo
    return times.getServerZoneinfo()
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\times.py", line 158, in getServerZoneinfo
    serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz')
  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 530, in simpleRequest
    raise splunk.AuthenticationFailed
AuthenticationFailed: [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - admin [01/Jul/2017:12:37:07.047 +0400] "GET /en-US/config?autoload=1 HTTP/1.1" 200 752 "http://10.6.1.114:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4" - 59575f330c40e8e6ab70 40ms
2017-07-01T12:37:07+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - admin [01/Jul/2017:12:37:07.318 +0400] "GET /en-US/config?autoload=1 HTTP/1.1" 200 752 "http://10.6.1.114:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4" - 59575f335140e8e7a6a0 13ms
2017-07-01T12:37:10+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated

Did anyone else experience this?

0 Karma

rapmancz
Explorer

I had similar issue. I removed settings.py and other settings.* files from bin folder, restarted splunk and the situation was the same.
Then I reinstalled the app (=downloaded the app file and uploaded it over web interface with checked update checkbox), restarted Splunk and after restart everything was OK again.

0 Karma

dvergnes_splunk
Splunk Employee
Splunk Employee

You probably need to remove settings.py, settings.pyo, settings.pyc from $SPLUNK_HOME/etc/apps/splunk_app_db_connect/bin. You can check http://docs.splunk.com/Documentation/DBX/3.1.0/DeployDBX/Checklist#Upgrade_an_existing_DB_Connect_de... for details. It is also mentioned in the release notes http://docs.splunk.com/Documentation/DBX/3.1.0/ReleaseNotes/Releasenotes.

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!