Connection Error with DBConnect to MySQL

[BryanScovill]

For some reason, after a VM event that caused a disruption to my Splunk's connection to downstream DBs, my MySQL connections will no longer connect. They report back "Could not create connection to database server" or ""Host 'XXX' is blocked because of many connection errors;"

Connections to Postgres DBs stayed just fine. And, oddly, we can create a valid MySQL connection via the same server's OS level MySQL client. Logs on the DB side don't even suggest a connection attempt when we try via DB Connect, although the error sort of contradicts that. Logs on the Splunk side don't yield anything useful. All of the searching I've done points at a network level issues, but that doesn't appear to be the case.

DB Connect and the MYSQL driver are all up to date and haven't been changed recently. Anyone have any thoughts on how to proceed?

Thanks.

[BryanScovill]

So, after a lot of digging and involving TAC and getting the case escalated we finally found a solution. It turns out my timelines were off and the root cause was an upgrade of the DB. Splunk's 5.1 driver was failing the handshake with MariaDB 8.0.17 so we went and manually upgrade the Connector/J to 8 and poof! Happy connections again.

The folks at TAC were telling me that the 8 driver was still in testing as part of the DB Connect package and that package wasn't ready yet. They thought the driver update had a good chance of helping so we tried it manually.

It was a real pain of a problem because the DB logs showed nothing at all unless we set debug to 4 and then only a handshake error not tied to any source. And handshake errors show up on other connections that are successful at that level of debug.

Thanks.

[jawaharas]

Try to increase the logging level and check the internal logs (splunk_app_db_connect*.log) to troubleshoot the issue.

Log configuration URL: http://splunk-host:8000/en-GB/app/splunk_app_db_connect/configuration#/settings/logconfiguration