Java Bridge not running

[redc]

I recently upgraded Java JRE to version 7 update 51 on the server running Splunk (Windows Server 2008 R2, SP1). After doing so, the Java Bridge stopped running. There don't appear to be any errors in either dbx.log or jbridge.log; however, what is in those two logs seems a little odd (see below).

These are my settings:

Java home: C:\Program Files (x86)\Java\jre7

JVM Commandline options: -Xmx256m -Dfile.encoding=UTF-8 -Duser.language=en -Duser.region=

I seem to recall the last time we upgraded Java JRE, we had to make some change to the commandline options, but I don't remember what that change was.

This is what's in the dbx.log (repeats roughly every 4.5 seconds):

2014-02-26 09:54:38.251 main:INFO:Splunkd - Initialized Splunkd REST service service=127.0.0.1:8089 sessionKey=0fc57988c8061f0326499d14eff6d710

This is what's in the jbridge.log (repeats roughly every 4.5 seconds with a different PID every time):

2014-02-26 09:54:45,332 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-02-26 09:54:45,332 INFO Started JavaBridgeServer PID=2808

Has anyone else run into this? If so, what did you do?

---

EDIT: additional log output after upgrading DB Connect to 1.1.3 and implementing workaround from http://answers.splunk.com/answers/105605/the-java-bridge-server-is-not-running-dbx-110

2014-02-27 08:43:11,065 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.thoughtworks.xstream.io.StreamException:  : Premature end of file.  at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:105)   at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:81)    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:904)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file.     at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)   at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)    at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:98)    ... 14 more 

2014-02-27 08:43:11,065 ERROR Command output: None

---

EDIT: NEW log output after uninstalling/reinstalling DB Connect:

2014-03-04 08:51:11,897 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file    at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119)     at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71)  at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51)    at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108)  at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.splunk.persistence.PersistentValueStoreException: File not found while trying to load persistent values from XML file   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:115)    at com.splunk.persistence.impl.XStreamStore.<init>(XStreamStore.java:49)    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at com.splunk.util.Utils$Reflection.instantiate(Utils.java:880)     at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117)     ... 4 more Caused by: java.io.FileNotFoundException: E:\Splunk\var\lib\splunk\persistentstorage\dbx\global\state.xml (Access is denied)     at java.io.FileInputStream.open(Native Method)  at java.io.FileInputStream.<init>(Unknown Source)   at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113)    ... 11 more 
2014-03-04 08:51:11,897 ERROR Command output: None
2014-03-04 08:51:13,644 INFO 10.110.68.254 - admin [04/Mar/2014:08:51:13.628 -0600] "GET /en-US/api/messages/index HTTP/1.1" 200 341 "http://iprdq-analytics.iproduction.com/en-US/manager/sourcemedia/data/ui/nav/default?uri=%2FservicesNS%2Fnobody%2Fsourcemedia%2Fdata%2Fui%2Fnav%2Fdefault&f_ns=sourcemedia&ns=sourcemedia&action=edit" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0" - 5315e861a0416beb8 16ms
2014-03-04 08:51:14,923 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-03-04 08:51:14,923 INFO Started JavaBridgeServer PID=1936

[KpiBuff]

Adding the local SYSTEM account to C:\Program Files\Splunk\var\lib\splunk\persistentstorage\dbx folder permissions resolved my issue.

[lukejadamec · Accepted solution]

We have a match. With an answer!
http://answers.splunk.com/answers/120422/dbconnect-stopped-functioning
Ain't dbconnect fun?

[redc]

Looks like when I reinstalled DB Connect, it only gave my user permission to read $SPLUNK_DB/persistentstorage/dbx (and therefore, everything underneath it). I changed the permissions to give SYSTEM access and that seems to have fixed the issue (at least temporarily).

Seems a little odd that this was working just fine after I did the uninstall/reinstall process but stopped working later...

[lukejadamec]

Easy tiger. That is essentially the exact same error. Did you step through the troubleshooting for the error found here?
http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Error_creating_PersistentValu...

[redc]

UGH! NO!

Problem's back, new error message (see second edit in original post).

This is occurring on both of my servers. One of them, I had to do a completely clean install (I did not restore the confs from the local folder), so I don't see how this could possibly still be a corruption issue.

Tried removing the database.conf file (per http://answers.splunk.com/answers/101479/db-connect-jbridge-wont-start ), but that didn't do it.

[redc]

That, combined with the forcible removal and reinstall of DB Connect (as described in the troubleshooting section linked to in that document) seems to have fixed the problem. At long last!

(Our data center experienced a major outage on Tuesday which cut off our virtual servers from their iSCSI drives, so I suspect that's what caused the state.xml files to become corrupted.)

Thank you for your patience and perseverance!

[redc]

Well, it's long and full of gobbledygook, but that's where it seems the resemblance ends. I'll post it in an answer below (it's way too long to go in a comment).

EDIT: I just edited it into the original post...

[lukejadamec]

That looks familiar. Does the jbridge error look like the one in this post?
http://answers.splunk.com/answers/77008/java-birdge-server-not-running-in-splunk-db-connect

[redc]

I am NOT, however, seeing the splunkd.log lines from that other Java Bridge thread anymore. This is all I see for "jbridge" in splunkd.log now:

02-27-2014 08:17:48.644 -0600 INFO  ExecProcessor - New scheduled exec process: python E:\Splunk\etc\apps\dbx\bin\jbridge_server.py
02-27-2014 08:17:48.644 -0600 INFO  ExecProcessor -     interval: 0 ms
02-27-2014 08:17:48.644 -0600 INFO  ExecProcessor - interval="0 01 * * *" is a valid cron schedule

[redc]

Yes, and even restarted Splunk for good measure.

Still seeing this in the dbx.log (absolutely nothing else but this repeatedly):

2014-02-27 08:31:33.109 main:INFO:Splunkd - Initialized Splunkd REST service service=127.0.0.1:8089 sessionKey=74557a4280bb537c02fd07b8a72e84f9

This in jbridge.log:

2014-02-27 08:32:05,667 INFO Checking for obsolete java libraries in E:\Splunk\etc\apps\dbx\bin\lib
2014-02-27 08:32:05,681 INFO Started JavaBridgeServer PID=2040
2014-02-27 08:32:06,759 ERROR Java process returned error code 1!
[this has "Premature end of file" errors in it, too]

[lukejadamec]

Did you open and save the settings from the GUI?
Any change to the errors in the dbx, jbridge, and splunkd logs?

[redc]

Ah-ha, okay.

I've implemented that __init__.py file solution, but ever since I upgraded DB Connect to v.1.1.3, the app won't load properly. It loads everything except the Java Bridge status, and then reloads the page, like it's stuck in some kind of loop when it tries to display the Java Bridge status.

[lukejadamec]

The `bin/spp folder is in the splunk install, not the java install. Here:
$SPLUNK_HOME/etc/apps/dbx/bin/spp/java/init.py file

[redc]

Tried to resave the setup settings and got:

Encountered the following error while trying to update: In handler 'localapps': Error while posting to url=/servicesNS/nobody/dbx/dbx/install/java

(I gotta give up for the night, I'll check back in in the morning.)

[redc]

Well, apparently, upgrading DB Connect and re-upgrading to Java 7u51 was a bad idea:

2014-02-26 18:26:37,380 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=E:\Splunk,SPLUNK_DB=E:\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. [bunch more gobbledygook]
2014-02-26 18:26:37,380 ERROR Command output: None

[redc]

I did find some jbridge logs in splunkd.log after restarting Splunk the last time, which match this:
http://answers.splunk.com/answers/105605/the-java-bridge-server-is-not-running-dbx-110

But...there's no \bin\spp folder in my Java install, so I can't implement their workaround.

I'm currently upgrading DB Connect to the latest version (was running 1.1.0).

[redc]

I've repeatedly tried resaving the settings (prior to making changes to them). The logs shown in the Answer for that thread do NOT appear in my logs, which seems to suggest that something isn't working right in the GUI.

[lukejadamec]

Read this post... and try saving your dbx settings from the gui. The first log matches yours.
http://answers.splunk.com/answers/102729/db-connect-error-in-upgrading-to-110-version

[redc]

Yes, but nothing unusual turning up in there, either.

I just rolled back the JRE update to 7u40 (which is what it was before I installed the 7u51 update) and it's still not working. We had a major data center outage yesterday that took down all our servers, now I'm wondering if that hosed something on this server...

This is the only weird thing I see in splunkd.log, but it's been there long before this java bridge issue arose, so I don't think it's related:

02-26-2014 17:53:17.179 -0600 ERROR DispatchThread - Failed to read runtime settings: File does not exist

[lukejadamec]

I have seen this java won't stay running thing before. I just can't remember exactly what I did to fix it. DB connect can be so quirky.
I see you've checked the dbx and jbridge log. Have you checked the splunkd log?