topic Re: Java Bridge not running in Dashboards & Visualizations

Java Bridge not running

redc — Wed, 26 Feb 2014 16:03:10 GMT

I recently upgraded Java JRE to version 7 update 51 on the server running Splunk (Windows Server 2008 R2, SP1). After doing so, the Java Bridge stopped running. There don't appear to be any errors in either dbx.log or jbridge.log; however, what is in those two logs seems a little odd (see below).

These are my settings:

Java home: C:\Program Files (x86)\Java\jre7

JVM Commandline options: -Xmx256m -Dfile.encoding=UTF-8 -Duser.language=en -Duser.region=

I seem to recall the last time we upgraded Java JRE, we had to make some change to the commandline options, but I don't remember what that change was.

This is what's in the dbx.log (repeats roughly every 4.5 seconds):

2014-02-26 09:54:38.251 main:INFO:Splunkd - Initialized Splunkd REST service service=127.0.0.1:8089 sessionKey=0fc57988c8061f0326499d14eff6d710

This is what's in the jbridge.log (repeats roughly every 4.5 seconds with a different PID every time):

2014-02-26 09:54:45,332 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-02-26 09:54:45,332 INFO Started JavaBridgeServer PID=2808

Has anyone else run into this? If so, what did you do?

EDIT: additional log output after upgrading DB Connect to 1.1.3 and implementing workaround from http://answers.splunk.com/answers/105605/the-java-bridge-server-is-not-running-dbx-110

2014-02-27 08:43:11,065 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.  at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:105)   at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:81)    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:904)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.     at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)   at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)    at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:98)    ... 14 more 

2014-02-27 08:43:11,065 ERROR Command output: None

EDIT: NEW log output after uninstalling/reinstalling DB Connect:

2014-03-04 08:51:11,897 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:115)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: java.io.FileNotFoundException: E:\Splunk\var\lib\splunk\persistentstorage\dbx\global\state.xml (Access is denied)     at java.io.FileInputStream.open(Native Method)  at java.io.FileInputStream.<init>(Unknown Source)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    ... 11 more 
2014-03-04 08:51:11,897 ERROR Command output: None
2014-03-04 08:51:13,644 INFO 10.110.68.254 - admin [04/Mar/2014:08:51:13.628 -0600] "GET /en-US/api/messages/index HTTP/1.1" 200 341 "http://iprdq-analytics.iproduction.com/en-US/manager/sourcemedia/data/ui/nav/default?uri=%2FservicesNS%2Fnobody%2Fsourcemedia%2Fdata%2Fui%2Fnav%2Fdefault&f_ns=sourcemedia&ns=sourcemedia&action=edit" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" - 5315e861a0416beb8 16ms
2014-03-04 08:51:14,923 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-03-04 08:51:14,923 INFO Started JavaBridgeServer PID=1936

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 16:52:50 GMT

I should mention, I went in to try resaving the database connections and got "Unknown error" when trying to "Fetch database names". But again, nothing in the logs to indicate what actually happened.

Re: Java Bridge not running

lukejadamec — Wed, 26 Feb 2014 17:08:48 GMT

Does the updated java bin contain a server folder?

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 17:10:21 GMT

No, just "client", "dtplugin", and "plugin2" folders (along with all the various .dll files).

Re: Java Bridge not running

lukejadamec — Wed, 26 Feb 2014 17:13:59 GMT

It won't run without the server part - something to do with the jbridgeserver I think.

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 17:31:47 GMT

Okay, so...what do I need to do to get it back? Downgrade Java?

Re: Java Bridge not running

linu1988 — Wed, 26 Feb 2014 17:51:21 GMT

Hello,
I am going to change my answer. Seems like you only have the run time environment installed. I would go for a fresh installation.

Download the Java SE from _http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html and install it
Backup the ..\etc\apps\dbx\local folder content. It's a known issue while upgrading the dbx app from exist versions.
delete the dbx folder in app folder
restart splunk
install the dbx app. it will ask for restart DON'T RESTART IMMEDIATELY. COPY THAT LOCAL FOLDER CONTENT INTO THE DBX LOCAL FOLDER.
Restart splunk now.
Try to configure the dbx App now.

let us know what are the result.

Thanks

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 23:16:42 GMT

Neither commandline option was effective. I don't have a JDK installed, only JRE.

C:\Program Files\Java\jre7

In watching the task manager, it looks like java.exe is not running consistently. It'll pop up on the list for a few seconds and then disappear again.

I'm not seeing anything in the logs that seems to indicate that it's recognizing the changes I'm making, nor any of the normal behavior I'd expect to see.

I've restarted Splunk and even tried restarting the server a couple of times, but still nothing.

EDIT: I tried installing a JDK (1.7.0_51), but it doesn't work, either.

Re: Java Bridge not running

lukejadamec — Wed, 26 Feb 2014 23:19:21 GMT

I use the JDK instead of the JRE, but I believe there are JRE's that contain the server module. The JRE inside the SDK will have it. Java 7 v51 JDK does work.

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 23:30:46 GMT

I just installed the JDK version 1.7.0_51 (latest build) and it does not work. I don't see a JDK version 7u51...?

Re: Java Bridge not running

lukejadamec — Wed, 26 Feb 2014 23:48:25 GMT

1.7 is long for 7.

Re: Java Bridge not running

lukejadamec — Wed, 26 Feb 2014 23:53:03 GMT

I have seen this java won't stay running thing before. I just can't remember exactly what I did to fix it. DB connect can be so quirky.
I see you've checked the dbx and jbridge log. Have you checked the splunkd log?

Re: Java Bridge not running

redc — Wed, 26 Feb 2014 23:56:42 GMT

Yes, but nothing unusual turning up in there, either.

I just rolled back the JRE update to 7u40 (which is what it was before I installed the 7u51 update) and it's still not working. We had a major data center outage yesterday that took down all our servers, now I'm wondering if that hosed something on this server...

This is the only weird thing I see in splunkd.log, but it's been there long before this java bridge issue arose, so I don't think it's related:

02-26-2014 17:53:17.179 -0600 ERROR DispatchThread - Failed to read runtime settings: File does not exist

Re: Java Bridge not running

lukejadamec — Thu, 27 Feb 2014 00:04:15 GMT

Read this post... and try saving your dbx settings from the gui. The first log matches yours.
http://answers.splunk.com/answers/102729/db-connect-error-in-upgrading-to-110-version

Re: Java Bridge not running

redc — Thu, 27 Feb 2014 00:10:30 GMT

I've repeatedly tried resaving the settings (prior to making changes to them). The logs shown in the Answer for that thread do NOT appear in my logs, which seems to suggest that something isn't working right in the GUI.

Re: Java Bridge not running

redc — Thu, 27 Feb 2014 00:17:35 GMT

I did find some jbridge logs in splunkd.log after restarting Splunk the last time, which match this:
http://answers.splunk.com/answers/105605/the-java-bridge-server-is-not-running-dbx-110

But...there's no \bin\spp folder in my Java install, so I can't implement their workaround.

I'm currently upgrading DB Connect to the latest version (was running 1.1.0).

Re: Java Bridge not running

redc — Thu, 27 Feb 2014 00:29:32 GMT

Well, apparently, upgrading DB Connect and re-upgrading to Java 7u51 was a bad idea:

2014-02-26 18:26:37,380 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. [bunch more gobbledygook]
2014-02-26 18:26:37,380 ERROR Command output: None

Re: Java Bridge not running

redc — Thu, 27 Feb 2014 00:31:03 GMT

Tried to resave the setup settings and got:

Encountered the following error while trying to update: In handler 'localapps': Error while posting to url=/servicesNS/nobody/dbx/dbx/install/java

(I gotta give up for the night, I'll check back in in the morning.)

Re: Java Bridge not running

lukejadamec — Mon, 28 Sep 2020 15:59:26 GMT

The `bin/spp folder is in the splunk install, not the java install. Here:
$SPLUNK_HOME/etc/apps/dbx/bin/spp/java/init.py file

Re: Java Bridge not running

redc — Thu, 27 Feb 2014 14:22:08 GMT

Ah-ha, okay.

I've implemented that __init__.py file solution, but ever since I upgraded DB Connect to v.1.1.3, the app won't load properly. It loads everything except the Java Bridge status, and then reloads the page, like it's stuck in some kind of loop when it tries to display the Java Bridge status.