JMS Messaging Modular Input: How to prevent indexing duplicate JMS messages coming from a JMS topic?

[ashleyherbert]

Hi,

We currently use the JMS Messaging Modular Input to retrieve JMS message from MQ queues, and the jms_ta runs on all 4 indexers in our indexer cluster.

We have a new requirement to retrieve messages from a Tibco EMS Topic, but if we use the same setup (the jms_ta running on 4 indexers), I'm assuming we will get multiple copies of the messages in Splunk as each of the indexers acts as a different topic subscriber. Is there any way for the jms_ta modules to know about each other to stop it receiving multiple copies? Or any suggestions on how to have highly available consumers of the JMS topic?

Thanks,
Ash

[Damien_Dallimor]

I'm assuming we will get multiple copies of the messages in Splunk as each of the indexers acts as a different topic subscriber

That is correct and is simply how topics works , 1 to many delivery.

There is no functionality for JMS Mod Input instances to know about other JMS Mod Input instances in order to co-ordinate an HA/Failover strategy and prevent duplicate message consumption off topics.

You would have to write some custom monitoring daemon (unless I have a better idea after a coffee)

1) setup (n) jms_ta stanzas , but only enable 1 (active) , the other (n) disabled (standby).
2) your monitoring daemon periodically checks that the active instance is ok.
3) if not ok , enable a standby instance (using the Splunk REST API)

[ashleyherbert]

Thanks for the quick response Damien, that's a better response than I'd be able to give before I've had my coffee!!

Yeah wanted to check if this had been considered before we started developing something new. I'm thinking it might be better to just have a forwarder running somewhere to retrieve the messages. It's not highly available, but at least it won't be affected by maintenance on the Splunk servers...