All Apps and Add-ons

Splunk DB Connect: Error in 'dbxquery' command: Invalid message received

burakcinar
Path Finder

Hello,

I'm getting that error after upgrading Splunk Enterprise v7.0 .. is there anyone that can help me ? : )

Thanks

Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

App Version 3.1.1
App Build 34

output of dbx2.log file

2017-02-23T14:02:30+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:22:30+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:35:08+0300 [INFO] [rpcstart.py], line 262: action=post_processing_for_rpc_server_termination rpc_start_pid=16840 rpc_server_pid=16987
2017-02-23T14:35:43+0300 [INFO] [rpcstart.py], line 366: action=run_rpc_start rpc_start_pid=3371 args=['/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py', '--scheme']
2017-02-23T14:36:03+0300 [INFO] [mi_session.py], line 38 : session updated
2017-02-23T14:36:05+0300 [INFO] [rpcstart.py], line 366: action=run_rpc_start rpc_start_pid=3831 args=['/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcstart.py']
2017-02-23T14:36:05+0300 [INFO] [rpcstart.py], line 125: action=start_to_run_rpc_server rpc_start_pid=3831
2017-02-23T14:36:08+0300 [INFO] [rpcstart.py], line 198: action=starting_up_rpc_server_with_command command="[u'/usr/java/jdk1.8.0_73/bin/java', u'-XX:+UseConcMarkSweepGC', '-classpath', '/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/mysql-connector-java-5.1.38-bin.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/postgresql-9.4.1208.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/ojdbc6.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/sqljdbc42.jar:/opt/splunk/etc/apps/splunk_app_db_connect/bin/lib/rpcserver-all.jar', '-DSPLUNK_HOME=/opt/splunk', 'com.splunk.dbx2.rpc.RPCServer', u'127.0.0.1:9998']"
2017-02-23T14:36:08+0300 [INFO] [rpcstart.py], line 243: action=rpc_server_process_is_launched rpc_start_pid=3831 rpc_server_pid=3942
2017-02-23T14:40:32+0300 [INFO] [rpcstart.py], line 262: action=post_processing_for_rpc_server_termination rpc_start_pid=3831 rpc_server_pid=3942

ammara
Explorer

I have just been dealing with the same issue after upgrading Splunk to version 7.01. I found the issue was resolved by simply upgrading the oracle database driver. I was using the old driver ojdbc6.jar, and after upgrading this to ojdbc7.jar everything worked fine again.

For instructions on how to install a new database driver and links to download the drivers see the below documentation page:
http://docs.splunk.com/Documentation/DBX/3.1.1/DeployDBX/Installdatabasedrivers
,I have just been dealing with the same problem after upgrading to Splunk 7.01. I found it was caused by the fact that I had the Oracle driver ojdbc6.jar. Once I updated this to ojdbc7.jar everything started working again.

Instructions for installing a new driver and links to where they can be downloaded can be found here:
http://docs.splunk.com/Documentation/DBX/3.1.1/DeployDBX/Installdatabasedrivers

0 Karma

earlhelms
Path Finder

I've been told that Db Connect 3.1.1 isn't compatible with Splunk version 7. You may be wasting your time trying to get it to work. I downgraded to Splunk 6 and DbConnect 3.1.1 seems fine so far.

cinchnetops
Explorer

The app site https://splunkbase.splunk.com/app/2686/ said it's compatible. I'm really hoping splunk can provide some guidance here.

earlhelms
Path Finder

Reference bug ticket DBX-4449

richgalloway
SplunkTrust
SplunkTrust

The DBX Release Notes for version 3.1.2 also lists this Known Issue

2018-01-09 DBX-4527 dbxquery command not able to function appropriately with Splunk Enterprise 7.0

---
If this reply helps you, Karma would be appreciated.
0 Karma

tmeader
Contributor

It says it NOW, yeah. They must've recently added it after numerous complaints (including mine) of there being no mention of this in the "Known Issues" section.

FYI, the actual fix for this is to go into the User's settings in Splunk and change their timezone to "System Default" (it has to be that setting). Then queries should work again. Would be nice if they also listed that workaround in the release notes.

Muryoutaisuu
Communicator

Just to be clear: Is it enough, if I delete all custom set tz settings from the users user-prefs.conf ? Or do I have to explicitly set tz to the same value as "System Default"?

thx in advance

0 Karma

RickCurry
Explorer

I too had this issue. I first tried downloading and engaging the ojdcb7.jar but that did not resolve the issue. So I tried changing to the Default Time Zone setting and BINGO! it works perfectly again.

Thank you to @tmeader for the tip that worked. Now to try v3.1.3. Woo-hoo!

0 Karma

prithivip
Observer

Hi Rick, May know exactly which file and location I need to edit to change Default Time Zone. I'm also facing the same problem tried with ojdbc7.jar and nothing helps.
I'm running DB Connect 3.1.4 and Splunk 7.2.5.1
Error :
Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

No information I can see in the log file.

0 Karma

RickCurry
Explorer

This setting is available in the Splunk Web UI by clicking on your ID in the menu bar and selecting "Preferences".

0 Karma

prithivip
Observer

Hi Rick, Thanks for quick response. I tried changing to Default Time Zone and vice versa and restarted my instance. Still I see the same error. Not sure where am I missing. Your suggestion is appreciated.

0 Karma

RickCurry
Explorer

prithivip,
Sorry to hear that did not resolve the issue for you. This is over 18 months old and I have long left behind the exact details to this. if you have not already done so, read over all of the responses and actions taken as there may actually be a combination of things that got us to the working result. My time, unfortunately, needs spent on my current activities.

Best wishes,
Rick

0 Karma

prithivip
Observer

Hi Rick, It worked now. Looks like my app might got corrupted. I'm not sure. When I tried by downloading again the same app in a fresh and it works. Anyway thanks for your time. I appreciate it.

0 Karma

tmeader
Contributor

To answer the question, it looks like like it is enough to complete remove it (delete the "tz" setting) from user-prefs.conf. That said, I got a notification for version 3.1.3 yesterday and just tried it out this morning. I can confirm this fixes the issue (at least for us) and we can now use custom timezones again while still having dbxquery working properly.

Good luck.

nuaraujo
Path Finder

Hello all, any update on this problem.

I have Splunk 7.0.1 running on Centos 6.6, with Splunk DB Connect. Any queries that I run always ends with Error in 'dbxquery' command: Invalid message received from external search command during setup, see search.log.

Thanks

0 Karma

markhvesta
Path Finder

We also have the Db Connect 3.1.1 running on 6 out of 7 hosts we have that are all running on Splunk Version 7 that do not encounter this issue. Only one host is seeing this issue.

0 Karma

cinchnetops
Explorer

Same issue.

I do see another java exception from splunk_app_db_connect_server.log
javax.management.RuntimeOperationsException: null
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:413)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at com.zaxxer.hikari.pool.PoolBase.registerMBeans(PoolBase.java:258)
at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:116)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:94)
at com.splunk.dbx.connector.ConnectorFactory.getConnectionFromPool(ConnectorFactory.java:181)
at com.splunk.dbx.connector.ConnectorFactory.getConnection(ConnectorFactory.java:169)
at com.splunk.dbx.connector.ConnectorFactory.create(ConnectorFactory.java:154)
at com.splunk.dbx.server.api.service.database.impl.DatabaseMetadataServiceImpl.getStatus(DatabaseMetadataServiceImpl.java:136)
at com.splunk.dbx.server.api.service.database.impl.DatabaseMetadataServiceImpl.getConnectionStatus(DatabaseMetadataServiceImpl.java:100)
at com.splunk.dbx.server.api.resource.ConnectionResource.getConnectionStatus(ConnectionResource.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:50)
at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:44)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at com.splunk.dbx.server.api.filter.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:51)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459)
at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:68)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Repository: cannot add mbean for pattern name com.zaxxer.hikari:type=PoolConfig (unnamed_pool_-1862066565_jdbc_postgresql//postgres.prd.cinchfinancial.com5432/mx?ssltrue&sslmoderequire&sslfactory_org.postgresql.ssl.NonValidatingFactory)

cinchnetops
Explorer

Same issue here. and just want to follow up that adding DEBUG doesn't seem to add any verbose error output from either splunk_app_db_connect_dbx.log nor splunk_app_db_connect_server.log

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...