All Apps and Add-ons

DBConnect stopped functioning

abonuccelli_spl
Splunk Employee
Splunk Employee

DBConnect was working ok, since few days I am seeing Java Bridge status 'not running'.
When checking $SPLUNK_HOME/var/log/splunk/jbridge.log

I can see this stacktrace all the time and JBridge failing to start:


2013-11-13 04:29:42,305 INFO Started JavaBridgeServer PID=3460
2013-11-13 04:29:42,305 DEBUG win32 handler registered
2013-11-13 04:29:43,164 ERROR Java process returned error code 1! Error: Initializing Splunk context... Environment: SplunkEnvironment{SPLUNK_HOME=C:\Program Files\Splunk,SPLUNK_DB=C:\Program Files\Splunk\var\lib\splunk} Configuring Log4j... [Fatal Error] :1:1: Premature end of file. Exception in thread "main" com.splunk.config.SplunkConfigurationException: Error creating PersistentValueStore type xstream: com.thoughtworks.xstream.io.StreamException: : Premature end of file. at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:119) at com.splunk.persistence.PersistentValueStoreFactory.createStore(PersistentValueStoreFactory.java:71) at com.splunk.persistence.PersistentValueStoreFactory.createGlobalStore(PersistentValueStoreFactory.java:51) at com.splunk.env.SplunkContext.initialize(SplunkContext.java:108) at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34) Caused by: com.thoughtworks.xstream.io.StreamException: : Premature end of file. at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:105) at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:81) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:904) at com.splunk.persistence.impl.XStreamStore.loadState(XStreamStore.java:113) at com.splunk.persistence.impl.XStreamStore.(XStreamStore.java:49) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.splunk.util.Utils$Reflection.instantiate(Utils.java:878) at com.splunk.persistence.PersistentValueStoreFactory.createStoreInstance(PersistentValueStoreFactory.java:117) ... 4 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source) at com.thoughtworks.xstream.io.xml.DomDriver.createReader(DomDriver.java:98) ... 14 more

How can I fix this?

1 Solution

abonuccelli_spl
Splunk Employee
Splunk Employee

This is due to corruption of state.xml file and subsequent failure to read it and start of JBridge Server embedded in DBConnect app.

This could happen on the global or input specific level:

$SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx

./global
./global/state.xml
./global/manifest.properties
./d842f2e1142e95ea0258a69682a7d591
./d842f2e1142e95ea0258a69682a7d591/state.xml
./d842f2e1142e95ea0258a69682a7d591/manifest.properties

state.xml files in folder named with long string refer to individual DBConnect inputs and preserve the value of the last record imported (watermark).

cat d842f2e1142e95ea0258a69682a7d591/state.xml



4371

state.xml files in folder named global, preserve value of last time DBConnect (JBridge Server) was started.

$ cat global/state.xml


1390840689919

If any of these XML files gets corrupted and becomes unparseable then parsing exception can happen:
com.thoughtworks.xstream.io.StreamException: : Premature end of file.
and JBridgeServer and DBConnect will stop working.

By renaming/removing the broken XML problem can be solved.

If removing any of the input specific XMLs , duplicate import of events might happen, as DBConnect will not have memory of what was the last ID/Timestamp indexed and will probably reindex from oldest timestamp/min(id).

View solution in original post

otman01
Communicator

thank you for your responses. the problem was fixed after review of the server configuration. I didin't have all priveleges into machine to get started the application.

0 Karma

otman01
Communicator

Hi every one,
I have a problem with splunk server, that it shows that The Java Bridge server is not running. I was looking for this directory $SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx to fix the issue but I didn't find it that means that I didn't find all the files that you talking about.

Have some ideas for my problem
Thank you .

abonuccelli_spl
Splunk Employee
Splunk Employee

This is due to corruption of state.xml file and subsequent failure to read it and start of JBridge Server embedded in DBConnect app.

This could happen on the global or input specific level:

$SPLUNK_HOME/var/lib/splunk/persistentstorage/dbx

./global
./global/state.xml
./global/manifest.properties
./d842f2e1142e95ea0258a69682a7d591
./d842f2e1142e95ea0258a69682a7d591/state.xml
./d842f2e1142e95ea0258a69682a7d591/manifest.properties

state.xml files in folder named with long string refer to individual DBConnect inputs and preserve the value of the last record imported (watermark).

cat d842f2e1142e95ea0258a69682a7d591/state.xml



4371

state.xml files in folder named global, preserve value of last time DBConnect (JBridge Server) was started.

$ cat global/state.xml


1390840689919

If any of these XML files gets corrupted and becomes unparseable then parsing exception can happen:
com.thoughtworks.xstream.io.StreamException: : Premature end of file.
and JBridgeServer and DBConnect will stop working.

By renaming/removing the broken XML problem can be solved.

If removing any of the input specific XMLs , duplicate import of events might happen, as DBConnect will not have memory of what was the last ID/Timestamp indexed and will probably reindex from oldest timestamp/min(id).

DataOrg
Builder

it worked after renaming the state.xml file. thanks 🙂

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Error_creating_PersistentValu... is the documentation link for this issue -- key to this answer is that you may get different types of error messages.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...