DB Connect configuration: Unable to save database inputs and slow search query

[wegscd]

Trying to get our freshly working DB Connect configured.

I am finding a problem in that I cannot save some new database inputs; the web GUI comes back with

Encountered the following error while trying to save: Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/search/dbx/dbmon: The read operation timed out',)

and the input does not get into app/dbx/local/inputs.conf.

I'm starting to notice a correlation between how slow the query I'm putting in is and above error occurring: if it's a slow query (say 30-60s because we are working against a large table), then I get the error.

Has anyone else seen this? Is there some kind of validation of the query that I can turn off so that I can add this input? Am I going to have to add the input into inputs.conf directly?

[gcoles]

FYI - you can reload the jbridge server by updating the JVM Command Line Options under the "Splunk DB Connect Configuration" dialog (http://your.splunk:8000/en-US/manager/dbx/apps/local/dbx/setup?action=edit). I typically just change the heap size between two equivalent sets of units (eg "-Xmx2048m" and "-Xmx2G") and save to restart the java bridge.

You can also reload the java bridge by hitting the debug/refresh endpoint as follows:

http://your.splunk:8000/en-US/debug/refresh?entity=admin/dbx-dblookups&entity=admin/dbx-dboutput

Verify that this worked by looking at the Java Bridge Status UI's uptime info.

[jkat54]

This made everything much quicker and less prone to timeouts, thanks!

[jonathan_octane]

I run into this constantly, and have found no reliable solution other than editing inputs.conf and restarting splunkd, which as you say, is not much of a solution. When I'm at a point where I simply can't afford the outage, sometimes I will keep banging away at the UI and the query will eventually save, but this issue needs attention for dbx to be taken seriously.

[pmdba]

My personal experience is that in order to get all of the parameters I need set correctly I almost always have to edit inputs.conf. The UI is getting better, but there are a couple of things it doesn't do. That said, if the query is timing out in the UI, it will probably still time out after you put it in inputs.conf.

You might consider whether there is any way to tune the query or the database to reduce the response time. You might also consider if if this is a "tail" type of input and the table has a lot of rows, not selecting historical data into Splunk; in such a case it might only be the initial query that takes so long. Once the historical data is indexed in Splunk performance might improve on its own.

[pmdba]

I assume there is a way to restart jbridge without restarting Splunk (as the UI seems to manage it somehow), but I've never found a way to do it from the command line. There's nothing in the documentation that I could find on the subject or on the "answers" site. Perhaps someone from Splunk Support could weigh in on this...

[brundl3fly]

This may help form the Splunk directory:

splunk cmd python $splunk_home/etc/apps/dbx/bin/reload.py databases

[wegscd]

the query is running fine (I can see it in dbx.log), but I just get the timeouts when I try to save the query (or changes) in the GUI.

Is there a way to just restart jbridge so it picks up my manually entered changes? restarting splunkd for every change to inputs.conf is expensive....

Optimization on the query is already done; it's an ugly one, not likely to get quicker.