All Apps and Add-ons

JMS Messaging Modular Input: How to prevent indexing duplicate JMS messages coming from a JMS topic?

ashleyherbert
Communicator

Hi,

We currently use the JMS Messaging Modular Input to retrieve JMS message from MQ queues, and the jms_ta runs on all 4 indexers in our indexer cluster.

We have a new requirement to retrieve messages from a Tibco EMS Topic, but if we use the same setup (the jms_ta running on 4 indexers), I'm assuming we will get multiple copies of the messages in Splunk as each of the indexers acts as a different topic subscriber. Is there any way for the jms_ta modules to know about each other to stop it receiving multiple copies? Or any suggestions on how to have highly available consumers of the JMS topic?

Thanks,
Ash

0 Karma

Damien_Dallimor
Ultra Champion

I'm assuming we will get multiple copies of the messages in Splunk as each of the indexers acts as a different topic subscriber

That is correct and is simply how topics works , 1 to many delivery.

There is no functionality for JMS Mod Input instances to know about other JMS Mod Input instances in order to co-ordinate an HA/Failover strategy and prevent duplicate message consumption off topics.

You would have to write some custom monitoring daemon (unless I have a better idea after a coffee)

1) setup (n) jms_ta stanzas , but only enable 1 (active) , the other (n) disabled (standby).
2) your monitoring daemon periodically checks that the active instance is ok.
3) if not ok , enable a standby instance (using the Splunk REST API)

0 Karma

ashleyherbert
Communicator

Thanks for the quick response Damien, that's a better response than I'd be able to give before I've had my coffee!!

Yeah wanted to check if this had been considered before we started developing something new. I'm thinking it might be better to just have a forwarder running somewhere to retrieve the messages. It's not highly available, but at least it won't be affected by maintenance on the Splunk servers...

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...