Java Access Bridge not running with Search Head Pooling

[bdevlinenova]

Recently installed the Splunk DB Connect app, made sure java was installed, followed what I could find on the requirements and documentation pages.

However when I launch the app it tells me that the Java Access Bridge is not running. If I try to add a database, it throws the error:

Encountered the following error while trying to update: In handler 'dbx-databases': Unexpected error "" from python handler: "Command com.splunk.bridge.cmd.Reload returned status code 47". See splunkd.log for more details.

I'm running on a Debian Squeeze server, and have tried installing multiple java packages, from what I could find 7u9 should have included the access bridge. Also tried installing the libaccess-bridge-java and libaccess-bridge-java-jni debian packages along with the sun-java6-jre/jdk.

It does seem to have some related files in /usr/:
./lib/jvm/java-6-openjdk/jre/lib/ext/libjava-access-bridge-jni.so
./lib/jni/libjava-access-bridge-jni.so.0
./lib/jni/libjava-access-bridge-jni.so
./lib/jni/libjava-access-bridge-jni.so.0.0.0
./share/doc/libaccess-bridge-java-jni
./share/doc/libaccess-bridge-java
./share/doc/libaccess-bridge-java/accessibility.properties

Followed some instructions from Oracle on setting it up, still does not see it however, and most of what I can find seems to be for the Windows version.

Does anyone have any helpful hints, or run into similar issues with this when getting it up and running?

[jcoates_splunk]

Hi,

we've just released DB Connect 1.1, which can now be installed on a search head pool.

app

search head pooling docs

Suspending the VM or changing the system clock will corrupt state and require a restart, we've documented that as a known issue.

Thanks,
Jack

[Dan · Accepted solution]

In summary, DB Connect does not support search head pooling.

With 1.0.7 there is a workaround you can apply to get it (basically) working. Note again this is not supported at this time:

1) DB Connect uses Splunk's secret key to encrypt configured passwords. Unfortunately Splunk instances in SHP don't share this secret key. As a workaround the same splunk.secret can manually be shared across all search heads in the pool.
2) Adding new database connections requires a restart on all other search heads in the pool.
3) Database inputs should not be used on SHP instances

[Dan]

Forgot you also need to adjust the scripted input. Add the following to apps/dbx/local/inputs.conf
[script://.\bin\jbridge_server.py]
index = _internal
sourcetype = dbx_jbridge
interval = 0
passAuth = admin
disabled = 0

By default, the stanza references the absolute path in $SPLUNK_HOME [script://$SPLUNK_HOME\etc\apps\dbx\bin\jbridge_server.py]

[dburns_splunk]

If you are seeing errors in your jbridge.log file like Error initiating Splunk context and Premature end of file messages, you may be seeing something similar to what I saw.

I run my apps on a VM under VMware Fusion and I have a habit of suspending VM's between uses (I do a lot of testing on various apps as a Splunk SE). My problem turned out to be corrupted storage, my guess, that is probably due to these frequent VM suspend operations.

I was instructed to do a recursive delete of the /global directory at this location:
$SPLUNK_DB/persistentstorage/dbx/global

When I did that, I restarted Splunk and the Java Bridge Server came right up.

Might be worth a try.

[ziegfried]

The Java Bridge in DB Connect is actually something different to the "Java Access Bridge" you are referring to. You can safely remove this package, since this is not required by DB Connect.

As for your problem regarding the Java Bridge not running:

1) Have you completed the setup page of the app? If not, in DB Connect, go to "Settings" in the navigation menu and Choose "Splunk DB Connect Configuration". You have to enter a valid Java/JRE installation directory (in your case probably /usr/lib/jvm/java-6-openjdk/jre) and then hit the "Save" button. If the JVM is for some reason not compatible with DB Connect, you should see an error message.

2) Please take a look into the jbridge.log and see if you can spot any errors. There is a saved search in the DB connect app that filters this log for errors (ie. Searches -> "Recent Java Bridge errors")

3) Take a look into DB Connect's debug log (Searches -> "Recent DB Connect errors")

[bdevlinenova]

So oddly enough, I finally figured out curl was not installed correctly and fixed that.

Now when I run the curl command you provided it sits there and runs, and I can go back into the app and it says that the java bridge is now running, however if I kill that command line process, it reverts back to not running and I can't get it started back up via the app. Now if I go in and try to resave the java seetings, and add in a database and validate I get:

12-12-2012 11:02:54.689 -0600 ERROR AdminManager - Unknown error while validating database connection

Can't find any other related errors.

[bdevlinenova]

File "/mnt/splunksearch/etc/apps/dbx/bin/spp/config.py", line 218, in postEditCallback
self.postModificationCallback()
File "/mnt/splunksearch/etc/apps/dbx/bin/rest_handler_databases.py", line 55, in postModificationCallback
executeBridgeCommand("com.splunk.bridge.cmd.Reload", args=["databases"], checkStatus=True)
File "/mnt/splunksearch/etc/apps/dbx/bin/spp/java/bridge.py", line 170, in executeBridgeCommand
raise JavaBridgeError("Command %s returned status code %s" % (cmd, ret))
JavaBridgeError: Command com.splunk.bridge.cmd.Reload returned status code 47

[bdevlinenova]

Got rather sidetracked but I tried redoing some stuff, and get this in the splunkd.log:

12-12-2012 09:44:20.759 -0600 ERROR AdminManager - Stack trace from python handler:
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 62, in init
hand.execute(info)
File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 449, in execute
if self.requestedAction == ACTION_EDIT: self.handleEdit(confInfo)
File "/mnt/splunksearch/etc/apps/dbx/bin/spp/config.py", line 215, in handleEdit
self.postEditCallback()

[bdevlinenova]

I then added in a symlink to the /opt/splunk/etc/apps directory and it moved on a bit farther, I'm going to continue diving in but I'm hoping you can pin point it faster.

This is now the resulting error:

Traceback (most recent call last):
File "/mnt/splunksearch/etc/apps/dbx/bin/jbridge_server.py", line 97, in
log.error("Error waiting for process: %s", e)
AttributeError: 'function' object has no attribute 'error'

[bdevlinenova]

So I think the problem is surrounding the fact that I do search head pooling and have set my app directory to be an nfs mount that is not in splunk home. It's under /mnt/splunksearch and I switched up the ending of the above curl command to call jbridge_server.py to reflect that.

Before I changed:

Traceback (most recent call last):
File "/mnt/splunksearch/etc/apps/dbx/bin/jbridge_server.py", line 26, in
for file in os.listdir(java.lib_path):
OSError: [Errno 2] No such file or directory: '/opt/splunk/etc/apps/dbx/bin/lib'

[ziegfried]

The script jbridge_server.py is expecting a valid REST session key in STDIN, so the this is fetched with curl, cleaned up and piped to the script call. You might need to adjust the username/password and the SPLUNK_HOME environment variable should be set.

It will not produce any output, but should see something in $SPLUNK_HOME/var/log/splunk/jbridge.log

[ziegfried]

There is a way to manually start the process:

$ curl -k -d 'username=admin' -d 'password=changeme' https://localhost:8089/services/auth/login 2>&1 | grep sessionKey | sed -e 's/.*<sessionKey>//g' | sed -e 's/<\/sessionKey>//g' | $SPLUNK_HOME/bin/splunk cmd python $SPLUNK_HOME/etc/apps/dbx/bin/jbridge_server.py

[ziegfried]

The JAVA_HOME environment variable actually has no affect on DB Connect.

You should see messages for the splunkd search above. Otherwise splunk is for some reason not running the scripted input. Please see if the following search brings up anything that might be relevant:

index=_internal sourcetype=splunkd ExecProcessor

Another question: What Splunk version are you using?

[bdevlinenova]

from local/inputs.conf
[script://$SPLUNK_HOME/etc/apps/dbx/bin/jbridge_server.py]
disabled = 0

No processes with jbridge or dbx running.

No iptables rules and everything is in ACCEPT mode.

The listed search also produced no results.

Verified that JAVA_HOME environment variable is set.

Is there any manual way of starting this?

[ziegfried]

Other suggestions:

• Verify that the scripted input [script:///$SPLUNK_HOME/etc/apps/dbx/bin/jbridge_server.py] is enabled
• See if splunkd's exec processor is emitting any error messages: Search: index=_internal sourcetype=splunkd jbridge_server

[ziegfried]

The setup containing just those Java settings is fine.

Could you try to see if the DB connect processes are running? Something like

$ pgrep -f dbx

or

$ ps -ef | grep jbridge

Also, please make sure that nothing (like iptables) is blocking TCP communication on the loopback interface (at least for the configured java bridge port - by default it's 17865).

[bdevlinenova]

1. I did do that a couple times prior, did it once more just to make sure and nothing. A thing to note is that in the install documentation it eludes to the fact that there should be multiple sections to the configure page, I only see the Java section. (I looked in the java.conf file and things seem to be set right there).
   1. and 3. I did the search for the last 24 hours, no results. Also tried turning up logging via the java.conf file and still no results.