Why is Splunk not closing Postgres connections with Splunk DB Connect 2?

[emalenfant]

I have a postgres user, named 'splunk' for this purpose - track what's happening.
It has a connection limit of 20 which should be more than enough.

Except, as soon as I even try to re-test a validation, all 20 connections are filled up.
This seems like a bug to me. It should be using only 1 connection, not a connection per query, or really, why the multiple queries.

I'm using the latest jdbc driver:
postgresql-9.4-1202.jdbc42.jar

[jcoates_splunk]

There's a couple of ways to get more modular inputs than you think...

1. make inputs run in more places than expected -- SHC and deployment server are popular ways to do this.
2. Run an OS shell that uses exciting and new variations in handling SIGTERMs (hi, /bin/dash).

[emalenfant]

Also: dbx2.log:

09/14/2015 17:50:45 [ERROR] [__init__.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 184, in dispatch
    methodOutput = method()
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 40, in handle_GET
    return self.get_catalogs(conn)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 206, in wrap_it
    return w.logIt (self, f, l, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 177, in logIt
    if err : raise err
AvroRemoteException: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [DEBUG] [health.py] Health logging is ON.
09/14/2015 17:50:46 [DEBUG] [rh_connections.py] Validating connection with URL [jdbc:postgresql://myserver:5432/db] failed: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [WARNING] [health.py] Validating connection with URL [jdbc:postgresql://myserver:5432/db] failed: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [ERROR] [rh_connections.py] exception Validating connection with URL [jdbc:postgresql://myserver:5432/db] failed: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [WARNING] [health.py] The user [eric] is not allowed to use health logger.
09/14/2015 17:50:46 [WARNING] [health.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [ERROR] [__init__.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 184, in dispatch
    methodOutput = method()
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 45, in handle_GET
    return self.get_schemas(conn, cat)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 206, in wrap_it
    return w.logIt (self, f, l, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 177, in logIt
    if err : raise err
AvroRemoteException: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [WARNING] [health.py] The user [admin] is not allowed to use health logger.
09/14/2015 17:50:46 [WARNING] [health.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [ERROR] [__init__.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 184, in dispatch
    methodOutput = method()
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 40, in handle_GET
    return self.get_catalogs(conn)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 206, in wrap_it
    return w.logIt (self, f, l, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 177, in logIt
    if err : raise err
AvroRemoteException: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [WARNING] [health.py] The user [admin] is not allowed to use health logger.
09/14/2015 17:50:46 [WARNING] [health.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:46 [ERROR] [__init__.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 184, in dispatch
    methodOutput = method()
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 51, in handle_GET
    return self.get_tables(conn, cat, schema)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 206, in wrap_it
    return w.logIt (self, f, l, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 177, in logIt
    if err : raise err
AvroRemoteException: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:47 [WARNING] [health.py] The user [admin] is not allowed to use health logger.
09/14/2015 17:50:47 [WARNING] [health.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
09/14/2015 17:50:47 [ERROR] [__init__.py] org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"
Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 184, in dispatch
    methodOutput = method()
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 40, in handle_GET
    return self.get_catalogs(conn)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 206, in wrap_it
    return w.logIt (self, f, l, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/health.py", line 177, in logIt
    if err : raise err
AvroRemoteException: org.postgresql.util.PSQLException:org.postgresql.util.PSQLException: FATAL: too many connections for role "splunk"