DBConnect stopped functioning

[abonuccelli_spl]

DBConnect was working ok, since few days I am seeing Java Bridge status 'not running'.
When checking $SPLUNK_HOME/var/log/splunk/jbridge.log

I can see this stacktrace all the time and JBridge failing to start:


2013-11-13 04:29:42,305 INFO Started JavaBridgeServer PID=3460
2013-11-13 04:29:42,305 DEBUG win32 handler registered
2013-11-13 04:29:43,164 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=C:\Program Files\Splunk,SPLUNK_DB=C:\Program Files\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.thoughtworks.xstream.io.StreamException: : Premature end of file. at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119) at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71) at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51) at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108) at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.thoughtworks.xstream.io.StreamException: : Premature end of file. at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:105) at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:81) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:904) at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113) at com.splunk.persistence.impl.XStreamStore.(XStreamStore.java:49) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.splunk.util.Utils$Reflection.instantiate(Utils.java:878) at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117) ... 4 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:98) ... 14 more


How can I fix this?

[otman01]

thank you for your responses. the problem was fixed after review of the server configuration. I didin't have all priveleges into machine to get started the application.

[otman01]

Hi every one,
I have a problem with splunk server, that it shows that The Java Bridge server is not running. I was looking for this directory $SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx to fix the issue but I didn't find it that means that I didn't find all the files that you talking about.

Have some ideas for my problem
Thank you .

[abonuccelli_spl · Accepted solution]

This is due to corruption of state.xml file and subsequent failure to read it and start of JBridge Server embedded in DBConnect app.

This could happen on the global or input specific level:

$SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx

./global
./global/state.xml
./global/manifest.properties
./d842f2e1142e95ea0258a69682a7d591
./d842f2e1142e95ea0258a69682a7d591/state.xml
./d842f2e1142e95ea0258a69682a7d591/manifest.properties

state.xml files in folder named with long string refer to individual DBConnect inputs and preserve the value of the last record imported (watermark).

cat d842f2e1142e95ea0258a69682a7d591/state.xml



4371

state.xml files in folder named global, preserve value of last time DBConnect (JBridge Server) was started.

$ cat global/state.xml


1390840689919

If any of these XML files gets corrupted and becomes unparseable then parsing exception can happen:
com.thoughtworks.xstream.io.StreamException: : Premature end of file.
and JBridgeServer and DBConnect will stop working.

By renaming/removing the broken XML problem can be solved.

If removing any of the input specific XMLs , duplicate import of events might happen, as DBConnect will not have memory of what was the last ID/Timestamp indexed and will probably reindex from oldest timestamp/min(id).

[DataOrg]

it worked after renaming the state.xml file. thanks 🙂

[jcoates_splunk]

http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Error_creating_PersistentValu... is the documentation link for this issue -- key to this answer is that you may get different types of error messages.