Java Bridge not running

[redc]

I recently upgraded Java JRE to version 7 update 51 on the server running Splunk (Windows Server 2008 R2, SP1). After doing so, the Java Bridge stopped running. There don't appear to be any errors in either dbx.log or jbridge.log; however, what is in those two logs seems a little odd (see below).

These are my settings:

Java home: C:\Program Files (x86)\Java\jre7

JVM Commandline options: -Xmx256m -Dfile.encoding=UTF-8 -Duser.language=en -Duser.region=

I seem to recall the last time we upgraded Java JRE, we had to make some change to the commandline options, but I don't remember what that change was.

This is what's in the dbx.log (repeats roughly every 4.5 seconds):

2014-02-26 09:54:38.251 main:INFO:Splunkd - Initialized Splunkd REST service service=127.0.0.1:8089 sessionKey=0fc57988c8061f0326499d14eff6d710

This is what's in the jbridge.log (repeats roughly every 4.5 seconds with a different PID every time):

2014-02-26 09:54:45,332 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-02-26 09:54:45,332 INFO Started JavaBridgeServer PID=2808

Has anyone else run into this? If so, what did you do?

---

EDIT: additional log output after upgrading DB Connect to 1.1.3 and implementing workaround from http://answers.splunk.com/answers/105605/the-java-bridge-server-is-not-running-dbx-110

2014-02-27 08:43:11,065 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.  at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:105)   at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:81)    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:904)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.     at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)   at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)    at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:98)    ... 14 more 

2014-02-27 08:43:11,065 ERROR Command output: None

---

EDIT: NEW log output after uninstalling/reinstalling DB Connect:

2014-03-04 08:51:11,897 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:115)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: java.io.FileNotFoundException: E:\Splunk\var\lib\splunk\persistentstorage\dbx\global\state.xml (Access is denied)     at java.io.FileInputStream.open(Native Method)  at java.io.FileInputStream.<init>(Unknown Source)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    ... 11 more 
2014-03-04 08:51:11,897 ERROR Command output: None
2014-03-04 08:51:13,644 INFO 10.110.68.254 - admin [04/Mar/2014:08:51:13.628 -0600] "GET /en-US/api/messages/index HTTP/1.1" 200 341 "http://iprdq-analytics.iproduction.com/en-US/manager/sourcemedia/data/ui/nav/default?uri=%2FservicesNS%2Fnobody%2Fsourcemedia%2Fdata%2Fui%2Fnav%2Fdefault&f_ns=sourcemedia&ns=sourcemedia&action=edit" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" - 5315e861a0416beb8 16ms
2014-03-04 08:51:14,923 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-03-04 08:51:14,923 INFO Started JavaBridgeServer PID=1936

[lukejadamec]

1.7 is long for 7.

[redc]

I just installed the JDK version 1.7.0_51 (latest build) and it does not work. I don't see a JDK version 7u51...?

[lukejadamec]

I use the JDK instead of the JRE, but I believe there are JRE's that contain the server module. The JRE inside the SDK will have it. Java 7 v51 JDK does work.

[linu1988]

Hello,
I am going to change my answer. Seems like you only have the run time environment installed. I would go for a fresh installation.

1. Download the Java SE from _http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html and install it
2. Backup the ..\etc\apps\dbx\local folder content. It's a known issue while upgrading the dbx app from exist versions.
3. delete the dbx folder in app folder
4. restart splunk
5. install the dbx app. it will ask for restart DON'T RESTART IMMEDIATELY. COPY THAT LOCAL FOLDER CONTENT INTO THE DBX LOCAL FOLDER.
6. Restart splunk now.
7. Try to configure the dbx App now.

let us know what are the result.

Thanks

[KpiBuff]

I am having the same woes over DB Connect's persistent storage.

I needed to alter the "bookmark" for where it begins reading SQL Audit logs. The Splunk Add-On for Sql Server could not determine the line log breaks. That was resolved, and the old indexed events deleted.

To have all the logs reloaded into splunk, I attempted editing state.xml with a date earlier than the first audit log entry. That did not work. Several other steps were taken with the persistentstorage\dbx\global folder, including backing up and deleting it.

Now, completely uninstalling and re-installing DB Connect and Java, following all the steps in the documentation and these recent posts, still will not recover the global folder.

Currently, "The Java Bridge server is not running ". Windows Task Manager shows Java(TM) Platform SE binary starting and stopping continuously. The jbridge.log says Unable to create directory C:\Program Files\Splunk\var\lib\splunk\persistentstorage\dbx\global.

The Java version that worked for me initially, and is currently installed:
server-jre-7u75-windows-x64.tar.gz

Using the above download link, it is very confusing determining the Java version to download. I gather 1.7 is the same as 7, and 7 will soon be unsupported by Oracle.

The DB Connect version is splunk-db-connect_117.tgz

This is all running on a single-server installation of SPLUNK on a Windows 2012 VM.

[linu1988]

is it the same server edition download link which i provided?

another step is set the java home, could you try this?

set %JAVA_HOME%="C:\Progr~1\Java\jdk..\jre"

[redc]

Neither commandline option was effective. I don't have a JDK installed, only JRE.

C:\Program Files\Java\jre7

In watching the task manager, it looks like java.exe is not running consistently. It'll pop up on the list for a few seconds and then disappear again.

I'm not seeing anything in the logs that seems to indicate that it's recognizing the changes I'm making, nor any of the normal behavior I'd expect to see.

I've restarted Splunk and even tried restarting the server a couple of times, but still nothing.

EDIT: I tried installing a JDK (1.7.0_51), but it doesn't work, either.

[redc]

Okay, so...what do I need to do to get it back? Downgrade Java?

[lukejadamec]

It won't run without the server part - something to do with the jbridgeserver I think.

[redc]

No, just "client", "dtplugin", and "plugin2" folders (along with all the various .dll files).

[lukejadamec]

Does the updated java bin contain a server folder?

[redc]

I should mention, I went in to try resaving the database connections and got "Unknown error" when trying to "Fetch database names". But again, nothing in the logs to indicate what actually happened.