Several other databases maintain connection through the app fine but one particular MySql database intermittently just seems to stop responding long enough that the 30 second timeout is hit. The queries I run normally do not take more than a second or two to finish, but several times throughout the day, dbconnect will hang when running a query and fail out with Exception(' java.sql.SQLTransientConnectionException: - Connection is not available, request timed out after 30000ms..',). "
When this starts to happen, I can't even validate the connection through the db app, if I open it in the explorer tab it will hang on validating and throw the same exact error message. I then have to come away for a while and come back when it randomly decides to start working again.
This database does not have any other reads or writes from any other sources occurring during the timeframe, the only other entity that uses this database is an app I have direct control over and I have it purposely not running any loads while I test out integrating the db with Splunk. I'm also running nothing scheduled through splunk in regards to this db, these are only single very simple ad hoc queries I am running, that take a second to return when things are "good".
In these times where splunk db connect has troubles, I have tested running the exact same queries through my own client, remotely from the searchheads, and they have no issue at all returning the queries within a second. Something specifically about the way dbconnect handles connection appears to be the issue but I cannot for the life of me figure out what. I've looked through https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Configureusertimeouts and this didn't really help me either.
I am kind of at the end of what I know to troubleshoot with this, I never have issues accessing this database through my app or through mysql clients, only through splunk do I ever seem to have issues. What else could I look into?