JMS Messaging Modular: Why are we seeing slow performance pulling messages from a Websphere MQ V8 queue via SSL?

[ashleyherbert]

Hi,

We're testing the JMS messaging modular input to pull messages from a Websphere MQ V8 queue, and have just got it working over SSL. However, we're seeing a massive performance degradation when connecting via SSL. Has anyone else got this working?

Using an unencrypted connection, we can pull close to 200 messages per second from the queue. Once we turn on SSL, that value drops below 20 messages per second.

Trying to find out what the bottleneck is, as it doesn't appear to be re-establishing connections every time. Are there any settings to improve this?

Thanks,
Ashley

[ashleyherbert · Accepted solution]

We've found the cause of this issue, and it's actually nothing to do with Splunk or the JMS Modular Input.
The problem is that we were using a VirtualBox VM for Splunk with the network configuration set to NAT. When we changed the network to 'Bridged' mode, the performance is good with both SSL & non-SSL. As soon as we switch NAT back on, the performance using SSL immediately drops by about 90%. Using non-SSL is also fine with both network modes. I've done a bit of googling to see if it's a known VirtualBox issue, but haven't been able to find anything. Good to know it's not an issue with the JMS Modular Input, thanks heaps for your help Damien!

[Damien_Dallimor]

You are not setting the JVM system properties correctly.
You dont need the "-D" , this is only need for passing system properties into a JVM from command line execution.

[ashleyherbert]

Thanks, yeah I originally had it without the "-D" but it didn't work either. I'll play around with this to get it going, but the most pressing issue is the performance. I'm going to try getting different versions of the client jars (newer & older) to see if there's any bugs in the version we're using. I'll also try different versions of Java on the client side. We've also been working with our MQ experts to review the config, and there doesn't appear to be anything on the Queue Manager or Queue that would cause this performance issue. Any other suggestions on things to try? Should I try setting up the Local handler that you have, would this behave any differently?

[Damien_Dallimor]

Yes , by all means try the local handler. Many people use this when going down the SSL route actually.

[Damien_Dallimor]

You can see the local handler and parameters that you can pass in here : https://gist.github.com/damiendallimore/b046a7e784775f7449d4

[Damien_Dallimor]

Well it is going to be something WAS specific , either in the client JMS drivers or on the Queue Manager side.

Can you share you inputs.conf setup ? version of WAS client jars being used ?

[ashleyherbert]

Hi Damien,
We're using the client jars from MQ V8.0.0.0 (not WAS. In fact, the jars were copied from the queue manager that we're connecting to). And I'm using JRE 1.8 to connect to it.

I also had issues trying to get the JVM System Properties to be picked up, so the keyStore setting isn't working. I've ended up just putting the trust cert into cacerts, but I'll need to change that (We'll need to use SSL MA once I've got this working correctly).

Below is my inputs.conf setup:

[jms://queue/:PAYMENTMSG]
browse_frequency = -1
browse_mode = all
browse_queue_only = 0
durable = 0
index = main
index_message_header = 0
index_message_properties = 0
init_mode = jndi
jms_connection_factory_name = PAYMON
jndi_initialcontext_factory = com.sun.jndi.fscontext.RefFSContextFactory
jndi_provider_url = file:/opt/splunk/indexer/etc/apps/jms_ta/mq/
sourcetype = mq
strip_newlines = 1
disabled = 0
jvm_system_properties = -Djavax.net.ssl.keyStore=/opt/splunk/indexer/etc/apps/jms_ta/mq/clientjks.jks,-Djavax.net.ssl.keyStorePassword=xxxxxxxx