DBX 3 running out of connections in pool

[ejharts2015]

Due to some database schema changes we're having to increase the timeouts on many of our mysql and redshift queries to our databases. As such it would appear we ran out of connectors in our connection pool (which I believe is default to 😎 and were thus unable to make new connections for subsequent runs.

Here is the error I see in the log:
2018-06-05 21:15:25.131 +0000 [dw-53 - GET /api/connections/awesome_db] ERROR com.splunk.dbx.connector.ConnectorFactory - action=failed_to_load_get_connection error=unnamed_pool_939597389_jdbc_mysql_//awesome_db - Connection is not available, request timed out after 30000ms. cause={}com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor182.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:341) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:779) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at com.splunk.dbx.service.driver.DelegatingDriver.connect(DelegatingDriver.java:23) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:430) at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:64) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:570) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:563) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.net.UnknownHostException: awesome_db at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:188) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:300) ... 24 common frames omitted

My question what impact would increasing the available connectors in the pool have on our splunk installation? And has anyone had success doing so. Thanks!

[mic1024]

Hi,
Did u ever found a fix for that problem?

[ejharts2015]

Nope. Sometimes it just does and after a while it clears up 😕

[tmuth_splunk]

Are you sure the HWF running DBX isn't overloaded on CPU or RAM? I would start there.

[ejharts2015]

Yes, DBX app is the only thing this host runs. There are no users or indexing or searching done on this box. We have < 10% CPU usage (average) and < 5 GB of ram usaged (average out of a total of 15 GB available).