Splunk DB Connect: Error in 'dbxquery' command: Invalid message received

[burakcinar]

Hello,

I'm getting that error after upgrading Splunk Enterprise v7.0 .. is there anyone that can help me ? : )

Thanks

Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

App Version 3.1.1
App Build 34

output of dbx2.log file

2017-02-23T14:02:30+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:22:30+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:35:08+0300 [INFO] [rpcstart.py], line 262: action=post_processing_for_rpc_server_termination rpc_start_pid=16840 rpc_server_pid=16987
2017-02-23T14:35:43+0300 [INFO] [rpcstart.py], line 366: action=run_rpc_start rpc_start_pid=3371 args=['/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py', '--scheme']
2017-02-23T14:36:03+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:36:05+0300 [INFO] [rpcstart.py], line 366: action=run_rpc_start rpc_start_pid=3831 args=['/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py']
2017-02-23T14:36:05+0300 [INFO] [rpcstart.py], line 125: action=start_to_run_rpc_server rpc_start_pid=3831
2017-02-23T14:36:08+0300 [INFO] [rpcstart.py], line 198: action=starting_up_rpc_server_with_command command="[u'/usr/java/jdk1.8.0_73/bin/java', u'-XX:+UseConcMarkSweepGC', '-classpath', '/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/mysql-connector-java-5.1.38-bin.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/postgresql-9.4.1208.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/ojdbc6.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/sqljdbc42.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/rpcserver-all.jar', '-DSPLUNK_HOME=/opt/splunk', 'com.splunk.dbx2.rpc.RPCServer', u'127.0.0.1:9998']"
2017-02-23T14:36:08+0300 [INFO] [rpcstart.py], line 243: action=rpc_server_process_is_launched rpc_start_pid=3831 rpc_server_pid=3942
2017-02-23T14:40:32+0300 [INFO] [rpcstart.py], line 262: action=post_processing_for_rpc_server_termination rpc_start_pid=3831 rpc_server_pid=3942

[ammara]

I have just been dealing with the same issue after upgrading Splunk to version 7.01. I found the issue was resolved by simply upgrading the oracle database driver. I was using the old driver ojdbc6.jar, and after upgrading this to ojdbc7.jar everything worked fine again.

For instructions on how to install a new database driver and links to download the drivers see the below documentation page:
http://docs.splunk.com/Documentation/DBX/3.1.1/DeployDBX/Installdatabasedrivers
,I have just been dealing with the same problem after upgrading to Splunk 7.01. I found it was caused by the fact that I had the Oracle driver ojdbc6.jar. Once I updated this to ojdbc7.jar everything started working again.

Instructions for installing a new driver and links to where they can be downloaded can be found here:
http://docs.splunk.com/Documentation/DBX/3.1.1/DeployDBX/Installdatabasedrivers

[earlhelms]

I've been told that Db Connect 3.1.1 isn't compatible with Splunk version 7. You may be wasting your time trying to get it to work. I downgraded to Splunk 6 and DbConnect 3.1.1 seems fine so far.

[cinchnetops]

The app site https://splunkbase.splunk.com/app/2686/ said it's compatible. I'm really hoping splunk can provide some guidance here.

[earlhelms]

Reference bug ticket DBX-4449

[richgalloway]

The DBX Release Notes for version 3.1.2 also lists this Known Issue

2018-01-09 DBX-4527 dbxquery command not able to function appropriately with Splunk Enterprise 7.0

[tmeader]

It says it NOW, yeah. They must've recently added it after numerous complaints (including mine) of there being no mention of this in the "Known Issues" section.

FYI, the actual fix for this is to go into the User's settings in Splunk and change their timezone to "System Default" (it has to be that setting). Then queries should work again. Would be nice if they also listed that workaround in the release notes.

[Muryoutaisuu]

Just to be clear: Is it enough, if I delete all custom set tz settings from the users user-prefs.conf ? Or do I have to explicitly set tz to the same value as "System Default"?

thx in advance

[RickCurry]

I too had this issue. I first tried downloading and engaging the ojdcb7.jar but that did not resolve the issue. So I tried changing to the Default Time Zone setting and BINGO! it works perfectly again.

Thank you to @tmeader for the tip that worked. Now to try v3.1.3. Woo-hoo!

[prithivip]

Hi Rick, May know exactly which file and location I need to edit to change Default Time Zone. I'm also facing the same problem tried with ojdbc7.jar and nothing helps.
I'm running DB Connect 3.1.4 and Splunk 7.2.5.1
Error :
Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

No information I can see in the log file.

[RickCurry]

This setting is available in the Splunk Web UI by clicking on your ID in the menu bar and selecting "Preferences".

[prithivip]

Hi Rick, Thanks for quick response. I tried changing to Default Time Zone and vice versa and restarted my instance. Still I see the same error. Not sure where am I missing. Your suggestion is appreciated.

[RickCurry]

prithivip,
Sorry to hear that did not resolve the issue for you. This is over 18 months old and I have long left behind the exact details to this. if you have not already done so, read over all of the responses and actions taken as there may actually be a combination of things that got us to the working result. My time, unfortunately, needs spent on my current activities.

Best wishes,
Rick

[prithivip]

Hi Rick, It worked now. Looks like my app might got corrupted. I'm not sure. When I tried by downloading again the same app in a fresh and it works. Anyway thanks for your time. I appreciate it.

[tmeader]

To answer the question, it looks like like it is enough to complete remove it (delete the "tz" setting) from user-prefs.conf. That said, I got a notification for version 3.1.3 yesterday and just tried it out this morning. I can confirm this fixes the issue (at least for us) and we can now use custom timezones again while still having dbxquery working properly.

Good luck.

[nuaraujo]

Hello all, any update on this problem.

I have Splunk 7.0.1 running on Centos 6.6, with Splunk DB Connect. Any queries that I run always ends with Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

Thanks

[markhvesta]

We also have the Db Connect 3.1.1 running on 6 out of 7 hosts we have that are all running on Splunk Version 7 that do not encounter this issue. Only one host is seeing this issue.

[cinchnetops]

Same issue.

I do see another java exception from splunk_app_db_connect_server.log
javax.management.RuntimeOperationsException: null
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:413)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at com.zaxxer.hikari.pool.PoolBase.registerMBeans(PoolBase.java:258)
at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:116)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:94)
at com.splunk.dbx.connector.ConnectorFactory.getConnectionFromPool(ConnectorFactory.java:181)
at com.splunk.dbx.connector.ConnectorFactory.getConnection(ConnectorFactory.java:169)
at com.splunk.dbx.connector.ConnectorFactory.create(ConnectorFactory.java:154)
at com.splunk.dbx.server.api.service.database.impl.DatabaseMetadataServiceImpl.getStatus(DatabaseMetadataServiceImpl.java:136)
at com.splunk.dbx.server.api.service.database.impl.DatabaseMetadataServiceImpl.getConnectionStatus(DatabaseMetadataServiceImpl.java:100)
at com.splunk.dbx.server.api.resource.ConnectionResource.getConnectionStatus(ConnectionResource.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:50)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:44)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at com.splunk.dbx.server.api.filter.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:68)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Repository: cannot add mbean for pattern name com.zaxxer.hikari:type=PoolConfig (unnamed_pool_-1862066565_jdbc_postgresql//postgres.prd.cinchfinancial.com5432/mx?ssltrue&sslmoderequire&sslfactory_org.postgresql.ssl.NonValidatingFactory)

[cinchnetops]

Same issue here. and just want to follow up that adding DEBUG doesn't seem to add any verbose error output from either splunk_app_db_connect_dbx.log nor splunk_app_db_connect_server.log