I've just upgraded Splunk DB Connect to 3.1.0 from 3.0.6 and I get the "DBX Server is not available, please make sure it is started and listening on 1025 port or consult documentation for details" error. (I set the port to 1025 from the default as part of trying to troubleshoot, but it did not work with the original port 9998 either.
In splunkd.log I see this:
07-01-2017 14:29:54.225 +0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\windows_x86_64\bin\server.exe"" action=task_server_start_failed error=javax.servlet.ServletException: io.dropwizard.jersey.setup.JerseyServletContainer-4ee5d723@fb567993==io.dropwizard.jersey.setup.JerseyServletContainer,1,false stack=org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:661)\\org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:419)\\org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:875)\\org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)\\org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)\\org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\com.codahale.metrics.jetty9.InstrumentedHandler.doStart(InstrumentedHandler.java:103)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:252)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)\\org.eclipse.jetty.server.Server.start(Server.java:411)\\org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)\\org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)\\org.eclipse.jetty.server.Server.doStart(Server.java:378)\\org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)\\io.dropwizard.cli.ServerCommand.run(ServerCommand.java:53)\\io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:44)\\io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)\\io.dropwizard.cli.Cli.run(Cli.java:75)\\io.dropwizard.Application.run(Application.java:79)\\com.splunk.dbx.server.bootstrap.TaskServerStart.startTaskServer(TaskServerStart.java:97)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:59)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:108)\\
In splunk_app_dbconnect_dbx.log I get these
after installing 3.1.0:
2017-07-01T09:23:33+0400 [ERROR] [admin.py], line 421: Failed to fetch DMC settings to verify status
2017-07-01T09:23:33+0400 [ERROR] [admin.py], line 422: [HTTP 404] https://127.0.0.1:8089/services/dmc-conf/settings/settings; [{'code': None, 'type': 'ERROR', 'text': 'Not Found'}]
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\controllers\admin.py", line 418, in _is_DMCDisabled
dmc_settings = en.getEntity(DMC_SETTINGS['url'], 'settings')
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\entity.py", line 249, in getEntity
serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 550, in simpleRequest
raise splunk.ResourceNotFound(uri, extendedMessages=extractMessages(body))
ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/dmc-conf/settings/settings; [{'code': None, 'type': 'ERROR', 'text': 'Not Found'}]
after a restart:
2017-07-01T12:36:56+0400 [ERROR] [root.py], line 663: DJANGO: There was an error starting:
2017-07-01T12:36:56+0400 [ERROR] [root.py], line 664: The SECRET_KEY setting must not be empty.
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 661, in run
configure_django(global_cfg)
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 738, in configure_django
if should_start_django() or FORCE_ENABLE_DJANGO:
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py", line 771, in should_start_django
return len(settings.DISCOVERED_APPS) > 0 and not isLite()
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 53, in __getattr__
self._setup(name)
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 48, in _setup
self._wrapped = Settings(settings_module)
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\django\conf\__init__.py", line 152, in __init__
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
ImproperlyConfigured: The SECRET_KEY setting must not be empty.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Bus STARTING
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Set handler for console events.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread '_TimeoutMonitor'.
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Serving on 127.0.0.1:8065
2017-07-01T12:36:56+0400 [INFO] [root.py], line 130: ENGINE: Bus STARTED
2017-07-01T12:36:57+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread 'Monitor'.
2017-07-01T12:36:57+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - - [01/Jul/2017:12:36:57.298 +0400] "HEAD /favicon.ico HTTP/1.1" 303 124 "" "Splunk/6.6.0 (Windows Server 8.1 Standard Edition; arch=x64)" - 59575f294c40e8e6a6d8 16ms
2017-07-01T12:37:07+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [root.py], line 130: ENGINE: Started monitor thread 'Monitor'.
2017-07-01T12:37:07+0400 [ERROR] [config.py], line 138: [HTTP 401] Client is not authenticated
Traceback (most recent call last):
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\config.py", line 136, in getServerZoneInfo
return times.getServerZoneinfo()
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\lib\times.py", line 158, in getServerZoneinfo
serverStatus, serverResp = splunk.rest.simpleRequest('/search/timeparser/tz')
File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.py", line 530, in simpleRequest
raise splunk.AuthenticationFailed
AuthenticationFailed: [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - admin [01/Jul/2017:12:37:07.047 +0400] "GET /en-US/config?autoload=1 HTTP/1.1" 200 752 "http://10.6.1.114:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4" - 59575f330c40e8e6ab70 40ms
2017-07-01T12:37:07+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated
2017-07-01T12:37:07+0400 [INFO] [customlogmanager.py], line 63 : 127.0.0.1 - admin [01/Jul/2017:12:37:07.318 +0400] "GET /en-US/config?autoload=1 HTTP/1.1" 200 752 "http://10.6.1.114:8000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4" - 59575f335140e8e7a6a0 13ms
2017-07-01T12:37:10+0400 [ERROR] [startup.py], line 104: Unable to read in product version information; [HTTP 401] Client is not authenticated
Did anyone else experience this?
I had similar issue. I removed settings.py and other settings.* files from bin folder, restarted splunk and the situation was the same.
Then I reinstalled the app (=downloaded the app file and uploaded it over web interface with checked update checkbox), restarted Splunk and after restart everything was OK again.
You probably need to remove settings.py, settings.pyo, settings.pyc from $SPLUNK_HOME/etc/apps/splunk_app_db_connect/bin. You can check http://docs.splunk.com/Documentation/DBX/3.1.0/DeployDBX/Checklist#Upgrade_an_existing_DB_Connect_de... for details. It is also mentioned in the release notes http://docs.splunk.com/Documentation/DBX/3.1.0/ReleaseNotes/Releasenotes.