Archive

DB Connect: After patching our Red Hat Enterprise Linux 6 OS and updating java.conf file, why are we receiving HTTP 401 errors?

Builder

We recently patched our operating system (Red Hat Enterprise Linux 6) and the Java JRE path changed. When we updated the java.conf file for DBX and restarted Splunk, we started seeing the following errors in the jbridge.log file:

ERROR Java process returned error code 1! Error: Initializing Splunk context... 
Environment: SplunkEnvironment{SPLUNK_HOME=/opt/splunk,SPLUNK_DB=/opt/splunk/var/lib/splunk} 
Configuring Log4j...
Exception in thread "main" com.splunk.config.SplunkConfigurationException: IO Error while reading configuration from Splunkd: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000    
at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:195)     
at com.splunk.config.rest.RESTAdapter.readConfig(RESTAdapter.java:203)
at com.splunk.config.cache.CachedConfigurationAdapter.readConfig(CachedConfigurationAdapter.java:32)    
at com.splunk.config.cache.CachedConfigurationAdapter.readStanza(CachedConfigurationAdapter.java:40)    
at com.splunk.env.SplunkContext.getConfigStanza(SplunkContext.java:313)
at com.splunk.env.SplunkContext.initialize(SplunkContext.java:128)  
at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:534) 
-at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1458)   
at java.security.AccessController.doPrivileged(Native Method)   
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1452)     
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1106)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)    
at com.splunk.rest.Response.getContent(Response.java:54)    
at com.splunk.rest.Response.readXML(Response.java:62)   
at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:193)     ... 
6 more
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000  
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1403)  
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)   
at com.splunk.rest.Response.<init>(Response.java:31)
at com.splunk.rest.Splunkd.request(Splunkd.java:213)    
at com.splunk.rest.Splunkd.request(Splunkd.java:98) 
... 7 more

We're using DB Connect 1.1.4. Does anyone have any idea what could have caused this? Everything was working fine before the OS was patched. Now the Java Bridge server will no longer load, and DB Connect is completely non-functional because it isn't able to communicate with the REST services (HTTP 401 is an "unauthorized" access error).

0 Karma
1 Solution

Builder

The answer finally turned out to be updating our Java configuration. We had originally been using the Java runtime included with the operating system. We had to install the latest Java 7 SE runtime from Oracle and redirect DBX to use that by updating the $SPLUNK/etc/apps/dbx/local/java.conf file.

View solution in original post

Builder

The answer finally turned out to be updating our Java configuration. We had originally been using the Java runtime included with the operating system. We had to install the latest Java 7 SE runtime from Oracle and redirect DBX to use that by updating the $SPLUNK/etc/apps/dbx/local/java.conf file.

View solution in original post

Explorer

I just fixed this issue. The answer was so simple and I found it in a post 3 years old!

http://answers.splunk.com/answers/68739/dbx-app-failed-to-connect-to-mssql-database-by-using-ssl-ssl...

Explorer

Exactly the same problem here so keen to get an answer on this. The only difference is that I'm running on Unbuntu 14.04. After applying a bunch of OS patches last night this is the only thing that isn't working. My jbridge.log file is showing identical errors to those originally posted here.

I'll post back if I get anywhere with this as I need to get this working ASAP.

0 Karma

Path Finder

We are having the same issue (just on a new install). I opened a suport case up today and will see what they say.

0 Karma

Communicator

Has anyone figured this one out? I'm running into the same issues.

0 Karma

Splunk Employee
Splunk Employee

Hi. Would you mind posting your local/inputs.conf stanzas? It could be an incorrect stanza in local/inputs.conf. See: http://docs.splunk.com/Documentation/DBX/1.1.4/DeployDBX/Troubleshoot#inputs.conf_stanzas_must_match....

Thanks

0 Karma

Splunk Employee
Splunk Employee

Please file a support ticket with a diag. The most common problem we've seen is javabridge not getting authentication token due to inputs.conf mismatch (the javabridge itself is started as a scripted input). If there is indeed a non-null sessionKey in the logs, this needs to be looked into.

0 Karma

Builder

I'd love to, but your support portal won't let me log in, and you don't provide a phone number or other contact info for support anywhere on your web site that I could find. I get the following error trying to access the support portal:

Login Error
Your login attempt using single sign-on with an identity provider certificate has failed. Please contact your salesforce.com administrator for more information.

0 Karma

Builder

None of our input stanza have changed; they were all working before OS patches were applied. It isn't just inputs that are failing - no part of the DBX user interface is working. The database browse and ad-hoc query (dbquery command) functions do not work, nor does the Save Settings screen. The Java Bridge server is down. Anything that from DBX that involves a REST call is failing with an HTTP 401 (Not Authorized) error. No database connection is even attempted.

0 Karma

Path Finder

Few things.
If you search pool mechanisam you have run authentication on each db for all nodes.

Or
Check the use roles for Dbx user some times that will also mess up

0 Karma

Builder

The error is from the REST server itself, no connection to the database is even attempted because the Java Bridge server is not functional.

0 Karma

Builder

We're also seeing errors like this in the jbridge.log file:

2014-10-27 11:34:50,576 INFO 127.0.0.1 - username [27/Oct/2014:11:34:50.551 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424090163 HTTP/1.1" 401 1396 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661a8d7fedec411190 25ms
2014-10-27 11:34:51,771 INFO 127.0.0.1 - username [27/Oct/2014:11:34:51.743 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424091376 HTTP/1.1" 401 1396 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661bbe7fedf8014610 27ms
2014-10-27 11:34:52,933 INFO 127.0.0.1 - username [27/Oct/2014:11:34:52.907 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424092530 HTTP/1.1" 401 1397 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661ce87fede8119450 26ms

Again, the "401" error seems to indicate some sort of permissions issue, but nothing has changed that we are aware of...

0 Karma

Path Finder

Patching messedup the java path so please find the java version and go to db connect up and give the correct java path
ore java.conf
[java]
home = /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
options = -Xmx2G -Dfile.encoding=UTF-8 -server -Duser.language=en -Duser.region=

pwd

/opt/splunk_nas/etc/apps/dbx/local

then reload java bridge it will fix the issue

0 Karma

Builder

Java path was the first thing we fixed. The REST server is refusing connections from DBX for some reason that we don't understand.

0 Karma

Splunk Employee
Splunk Employee

Another possibility: Check the dbx.log for "REST keep-alive failed" error. If this turns up restart Splunk. You also might have a stale "state.xml" file. Fix is to remove any stale state.xml files. See: http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Java_bridge_log_file_settings

0 Karma

Builder

We tried this too. There is some sort of permissions issue with the REST server now - it is refusing to accept connections from DBX. Don't know why.

0 Karma

Splunk Employee
Splunk Employee

Hi. Did you update the JAVA HOME path inside DB Connect 1.1.4. This is a common oversight when upgrading JRE. Inside DB Connect, go to Settings > Splunk DB Connect Configuration. Then update the Java Home with the new path. For more info see:
http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Java_Bridge_Server_does_not_w...

Hope this helps

0 Karma

Builder

Yes, we had already updated the Java Home. Had to do it manually in the java.conf file because DB Connect can't use REST calls from the UI to do anything. Before we updated the Java Home we couldn't even get this far - no Java Bridge server and no errors in the logs at all. Now we at least get these error messages.