All Apps and Add-ons

DBX 3 running out of connections in pool

ejharts2015
Communicator

Due to some database schema changes we're having to increase the timeouts on many of our mysql and redshift queries to our databases. As such it would appear we ran out of connectors in our connection pool (which I believe is default to 😎 and were thus unable to make new connections for subsequent runs.

Here is the error I see in the log:
2018-06-05 21:15:25.131 +0000 [dw-53 - GET /api/connections/awesome_db] ERROR com.splunk.dbx.connector.ConnectorFactory - action=failed_to_load_get_connection error=unnamed_pool_939597389_jdbc_mysql_//awesome_db - Connection is not available, request timed out after 30000ms. cause={}com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor182.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:341) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:779) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at com.splunk.dbx.service.driver.DelegatingDriver.connect(DelegatingDriver.java:23) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:430) at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:64) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:570) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:563) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.net.UnknownHostException: awesome_db at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:188) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:300) ... 24 common frames omitted

My question what impact would increasing the available connectors in the pool have on our splunk installation? And has anyone had success doing so. Thanks!

0 Karma

mic1024
Path Finder

Hi,
Did u ever found a fix for that problem?

0 Karma

ejharts2015
Communicator

Nope. Sometimes it just does and after a while it clears up 😕

0 Karma

tmuth_splunk
Splunk Employee
Splunk Employee

Are you sure the HWF running DBX isn't overloaded on CPU or RAM? I would start there.

0 Karma

ejharts2015
Communicator

Yes, DBX app is the only thing this host runs. There are no users or indexing or searching done on this box. We have < 10% CPU usage (average) and < 5 GB of ram usaged (average out of a total of 15 GB available).

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!