DB Connect: After patching our Red Hat Enterprise Linux 6 OS and updating java.conf file, why are we receiving HTTP 401 errors?

[pmdba]

We recently patched our operating system (Red Hat Enterprise Linux 6) and the Java JRE path changed. When we updated the java.conf file for DBX and restarted Splunk, we started seeing the following errors in the jbridge.log file:

ERROR Java process returned error code 1! Error: Initializing Splunk context... 
Environment: SplunkEnvironment{SPLUNK_HOME=/opt/splunk,SPLUNK_DB=/opt/splunk/var/lib/splunk} 
Configuring Log4j...
Exception in thread "main" com.splunk.config.SplunkConfigurationException: IO Error while reading configuration from Splunkd: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000    
at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:195)     
at com.splunk.config.rest.RESTAdapter.readConfig(RESTAdapter.java:203)
at com.splunk.config.cache.CachedConfigurationAdapter.readConfig(CachedConfigurationAdapter.java:32)    
at com.splunk.config.cache.CachedConfigurationAdapter.readStanza(CachedConfigurationAdapter.java:40)    
at com.splunk.env.SplunkContext.getConfigStanza(SplunkContext.java:313)
at com.splunk.env.SplunkContext.initialize(SplunkContext.java:128)  
at com.splunk.bridge.JavaBridgeServer.main(JavaBridgeServer.java:34)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:534) 
-at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1458)   
at java.security.AccessController.doPrivileged(Native Method)   
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1452)     
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1106)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)    
at com.splunk.rest.Response.getContent(Response.java:54)    
at com.splunk.rest.Response.readXML(Response.java:62)   
at com.splunk.config.rest.RESTAdapter.request(RESTAdapter.java:193)     ... 
6 more
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://127.0.0.1:8089/servicesNS/nobody/-/admin/conf-java?count=10000  
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1403)  
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)   
at com.splunk.rest.Response.<init>(Response.java:31)
at com.splunk.rest.Splunkd.request(Splunkd.java:213)    
at com.splunk.rest.Splunkd.request(Splunkd.java:98) 
... 7 more

We're using DB Connect 1.1.4. Does anyone have any idea what could have caused this? Everything was working fine before the OS was patched. Now the Java Bridge server will no longer load, and DB Connect is completely non-functional because it isn't able to communicate with the REST services (HTTP 401 is an "unauthorized" access error).

[pmdba · Accepted solution]

The answer finally turned out to be updating our Java configuration. We had originally been using the Java runtime included with the operating system. We had to install the latest Java 7 SE runtime from Oracle and redirect DBX to use that by updating the $SPLUNK/etc/apps/dbx/local/java.conf file.

[damuscat]

I just fixed this issue. The answer was so simple and I found it in a post 3 years old!

http://answers.splunk.com/answers/68739/dbx-app-failed-to-connect-to-mssql-database-by-using-ssl-ssl...

[damuscat]

Exactly the same problem here so keen to get an answer on this. The only difference is that I'm running on Unbuntu 14.04. After applying a bunch of OS patches last night this is the only thing that isn't working. My jbridge.log file is showing identical errors to those originally posted here.

I'll post back if I get anywhere with this as I need to get this working ASAP.

[rsolutions]

We are having the same issue (just on a new install). I opened a suport case up today and will see what they say.

[mcm10285]

Has anyone figured this one out? I'm running into the same issues.

[sroback_splunk]

Hi. Would you mind posting your local/inputs.conf stanzas? It could be an incorrect stanza in local/inputs.conf. See: http://docs.splunk.com/Documentation/DBX/1.1.4/DeployDBX/Troubleshoot#inputs.conf_stanzas_must_match....

Thanks

[rroussev_splunk]

Please file a support ticket with a diag. The most common problem we've seen is javabridge not getting authentication token due to inputs.conf mismatch (the javabridge itself is started as a scripted input). If there is indeed a non-null sessionKey in the logs, this needs to be looked into.

[pmdba]

I'd love to, but your support portal won't let me log in, and you don't provide a phone number or other contact info for support anywhere on your web site that I could find. I get the following error trying to access the support portal:

Login Error
Your login attempt using single sign-on with an identity provider certificate has failed. Please contact your salesforce.com administrator for more information.

[pmdba]

None of our input stanza have changed; they were all working before OS patches were applied. It isn't just inputs that are failing - no part of the DBX user interface is working. The database browse and ad-hoc query (dbquery command) functions do not work, nor does the Save Settings screen. The Java Bridge server is down. Anything that from DBX that involves a REST call is failing with an HTTP 401 (Not Authorized) error. No database connection is even attempted.

[bkondakindi]

Few things.
If you search pool mechanisam you have run authentication on each db for all nodes.

Or
Check the use roles for Dbx user some times that will also mess up

[pmdba]

The error is from the REST server itself, no connection to the database is even attempted because the Java Bridge server is not functional.

[pmdba]

We're also seeing errors like this in the jbridge.log file:

2014-10-27 11:34:50,576 INFO 127.0.0.1 - username [27/Oct/2014:11:34:50.551 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424090163 HTTP/1.1" 401 1396 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661a8d7fedec411190 25ms
2014-10-27 11:34:51,771 INFO 127.0.0.1 - username [27/Oct/2014:11:34:51.743 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424091376 HTTP/1.1" 401 1396 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661bbe7fedf8014610 27ms
2014-10-27 11:34:52,933 INFO 127.0.0.1 - username [27/Oct/2014:11:34:52.907 -0400] "GET /en-US/custom/dbx/dbx/status?nocache=1414424092530 HTTP/1.1" 401 1397 "https://splunk/en-US/app/dbx/home" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0" - 544e661ce87fede8119450 26ms

Again, the "401" error seems to indicate some sort of permissions issue, but nothing has changed that we are aware of...

[bkondakindi]

Patching messedup the java path so please find the java version and go to db connect up and give the correct java path
ore java.conf
[java]
home = /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.51.x86_64/jre
options = -Xmx2G -Dfile.encoding=UTF-8 -server -Duser.language=en -Duser.region=

pwd

/opt/splunk_nas/etc/apps/dbx/local

then reload java bridge it will fix the issue

[pmdba]

Java path was the first thing we fixed. The REST server is refusing connections from DBX for some reason that we don't understand.

[sroback_splunk]

Another possibility: Check the dbx.log for "REST keep-alive failed" error. If this turns up restart Splunk. You also might have a stale "state.xml" file. Fix is to remove any stale state.xml files. See: http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Java_bridge_log_file_settings

[pmdba]

We tried this too. There is some sort of permissions issue with the REST server now - it is refusing to accept connections from DBX. Don't know why.

[sroback_splunk]

Hi. Did you update the JAVA HOME path inside DB Connect 1.1.4. This is a common oversight when upgrading JRE. Inside DB Connect, go to Settings > Splunk DB Connect Configuration. Then update the Java Home with the new path. For more info see:
http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Troubleshoot#Java_Bridge_Server_does_not_w...

Hope this helps

[pmdba]

Yes, we had already updated the Java Home. Had to do it manually in the java.conf file because DB Connect can't use REST calls from the UI to do anything. Before we updated the Java Home we couldn't even get this far - no Java Bridge server and no errors in the logs at all. Now we at least get these error messages.