multiple problems using Splunk java sdk 1.1.0 in weblogic 3.5.0

[preben12]

Hi

I want to test publishing events to Splunk from a webapp deployed on our Weblogic app server, but are running into several problems using the SDK.
First error using plain sdk:
Caused By: java.lang.IllegalArgumentException: value is null
at weblogic.net.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:776)
at com.splunk.HttpService.send(HttpService.java:293)
at com.splunk.Service.send(Service.java:1211)
at com.splunk.HttpService.post(HttpService.java:212)
at com.splunk.Service.login(Service.java:1044)
Truncated. see log file for complete stacktrace

After fixing the HttpService to include null check and generating a client cert from the splunk server -> importing it into Weblogics truststore the next error surfaces.

Caused By: java.lang.RuntimeException: [Security:090504]Certificate chain received from bamtst01 - 10.101.17.63 failed hostname verification check. Certificate contained SplunkServerDefaultCert but check expected bamtst01
at com.splunk.HttpService.send(HttpService.java:365)
at com.splunk.Service.send(Service.java:1211)
at com.splunk.HttpService.post(HttpService.java:239)
at com.splunk.Service.login(Service.java:1044)
at com.splunk.Service.login(Service.java:1024)
Truncated. see log file for complete stacktrace
Caused By: javax.net.ssl.SSLKeyException: [Security:090504]Certificate chain received from bamtst01 - 10.101.17.63 failed hostname verification check. Certificate contained SplunkServerDefaultCert but check expected bamtst01
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireException(Unknown Source)
at com.certicom.tls.interfaceimpl.TLSConnectionImpl.fireAlertSent(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)
at com.certicom.tls.record.handshake.HandshakeHandler.fireAlert(Unknown Source)
at com.certicom.tls.record.handshake.ClientStateReceivedServerHello.handle(Unknown Source)
Truncated. see log file for complete stacktrace

I have to turn hostname verification off in Weblogic 😞 to get further.

Next error is ->
Caused By: javax.xml.stream.XMLStreamException: next() may not be called when XMLStreamReader is at END_DOCUMENT
at weblogic.xml.stax.XMLStreamReaderBase.next(XMLStreamReaderBase.java:234)
at com.splunk.AtomObject.scan(AtomObject.java:193)
at com.splunk.AtomObject.load(AtomObject.java:128)
at com.splunk.AtomFeed.parse(AtomFeed.java:80)
at com.splunk.AtomFeed.parseStream(AtomFeed.java:59)
Truncated. see log file for complete stacktrace

That how far i got.
Seems to be some kind of Stax class loading issue. ???????

I saw a similar issue described at http://splunk-base.splunk.com/answers/68748/error-when-run-java-sdk-next-may-not-be-called-when-xmls.... Any solution for this ??

From looking at the sdk it seems to me that the client sdk should support client certificates to establish a trusted SSL connection out of the box. I'm not sure the trustAll truststore setup is recommended for production use.

Secondary is it possible to use pure http to connect to a splunk server if you don't need ssl, and if so on wich port can you connect ?

btw. I have created a support case of this CASE 117183.

Best Regards
Preben

[preben12]

Hi Kelvin

Afair I used the json parser instead to bypass wls xml libs in classpath. You can see some of the code here https://github.com/apache/camel/blob/master/components/camel-splunk/src/main/java/org/apache/camel/c...
where the json reader is used.

[lsmkelvin]

Thanks Preben12.

I got this exception same as you, how you fix it?
Caused By: javax.xml.stream.XMLStreamException: next() may not be called when XMLStreamReader is at END_DOCUMENT
at weblogic.xml.stax.XMLStreamReaderBase.next(XMLStreamReaderBase.java:234)
at com.splunk.AtomObject.scan(AtomObject.java:193)
at com.splunk.AtomObject.load(AtomObject.java:128)
at com.splunk.AtomFeed.parse(AtomFeed.java:80)
at com.splunk.AtomFeed.parseStream(AtomFeed.java:59)
Truncated. see log file for complete stacktrace

Thanks
Kelvin

[preben12]

yeah - was fixed in the splunk sdk afair

[lsmkelvin]

Hi Preben,

How you fix the below problem?

Caused By: java.lang.IllegalArgumentException: value is null
at weblogic.net.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:776)
at com.splunk.HttpService.send(HttpService.java:293)
at com.splunk.Service.send(Service.java:1211)
at com.splunk.HttpService.post(HttpService.java:212)
at com.splunk.Service.login(Service.java:1044)
Truncated. see log file for complete stacktrace

Best regards,
Kelvin

[preben12]

Yeah - I have upgraded to 5.0.3 and it resolved the reader issue

[preben12]

An update so far.
I made it work login and posting of events with a small patch in the Splunk jdk. See patch https://github.com/splunk/splunk-sdk-java/pull/45. The main fix forces the use of Suns https Handler.
I still got some issues reading events from Splunk when running in weblogic and using Xml streams. I'll get back with details when i'm back at work and have a running Weblogic.

I thought I could bypass that using ResultsReaderJson instead of ResultsReaderXml, by changing the output mode and changing the reader to ResultReaderJson. But the Json reader bloks on getNextEvent compared to the xml reader when there is no data to read from the stream (0 events) resulting in a not responding thread.
I could possible work around that by checking on the event count and only read if the count > 0, but that not doable in realtime mode I guess since count is not known on beforehand.

[alexniranjan]

I am on Splunk Enterprise 6.5 and Splunk JDK 1.6. I am still facing this issue while creating the connection. This URL is not accessible anymore - https://github.com/splunk/splunk-sdk-java/pull/45.

Do you have a workaround or another location for this fix?

[andras_kerekes]

What I did in this case is that used InputStream.mark() and reset() to peek into the result stream to see if it has no results. But it seems that in Splunk 5.0.3 the returned String will not put the ResultReaderJson into infinite loop, if you can upgrade this will solve it.

[preben12]

Hi Damien
Yeah turning on -DUseSunHttpHandler=true will make it work. The only thing that is a bit worrying is that this is a global option that will affect everything running on the platform. Chances are that i might cause side effects on running apps. Thats why i wanna try to avoid it.
On another project I have code wise forced weblogic to use sun's https handler when constructing the URL while importing the full package+class - This works on Weblogic
Code sample :
// force Weblogic to use suns https handler.
new URL(protocol, host, port, path, new sun.net.www.protocol.https.Handler());

[Damien_Dallimor]

Try out these JVM startup params and see if this lets you create a simple https connection.

http://weblogic-wonders.com/weblogic/2010/11/09/enforce-weblogic-to-use-sun-ssl-implementation-rathe...

[preben12]

You can find the code here https://github.com/pax95/camel-splunk
and where the splunk connection is created ->
https://github.com/pax95/camel-splunk/blob/master/src/main/java/org/apache/camel/component/splunk/Sp...

I have imported the certificate i got when connecting to https://myhost:8089/ saved the certificate and imported it into Weblogic. Is there any other certificate from Splunk that I should use instead ?

[preben12]

Nope - I just use the submit request on port 8089 using https - not tcp-ssl.
I haven't even gotten to send any data yet. Just trying to establish a connection.

[preben12]

Sorry Weblogic version is 10.3.5

[jonuwz]

Post your code. Are you using submit / attach / attachWith ?

Have you used this as a guideline ? link

Also, the cert you've imported has a CN of "DefaultSplunkCertificate" the HTTPS connection will not allow that because it doesn't match the hostname you are connecting to.