topic Re: How do handle quotes in JSON when receiving AMQP messages in All Apps and Add-ons

How do handle quotes in JSON when receiving AMQP messages

marcusnilssonmr — Mon, 28 Sep 2020 19:02:00 GMT

I am using Splunk AMQP Messaging Input with RabbitMQ. When posting a message with JSON content, I receive the following event in Splunk:

Fri Feb 20 11:57:56 CET 2015 name="amqp_msg_received" event_id="5" msg_queue="splunk_consume" msg_exchange="x" msg_body="{"x": 5, "y": 5}"

The problem is that since the json contains ", Splunk interprets msg_body as being "{", since the quotes are not escaped. What is the best way to handle this?

Re: How do handle quotes in JSON when receiving AMQP messages

Damien_Dallimor — Fri, 20 Feb 2015 20:11:33 GMT

Probably simplest to just do a custom field extraction in Splunk Web.
Follow the "Extract New Fields" link.

Re: How do handle quotes in JSON when receiving AMQP messages

marcusnilssonmr — Sun, 07 Jun 2015 08:31:26 GMT

Can we get the modular input to escape properly? Then all that is needed is a kvmodr that handles escaping.

Re: How do handle quotes in JSON when receiving AMQP messages

Damien_Dallimor — Mon, 28 Sep 2020 20:08:46 GMT

So when I write Modular Inputs that can conceivably receive any type of payload once they are being used in the wild , I have to employ an architecture that ships with a default message handler (the format you see above) but also allows the end user to plugin their own message handler to perform whatever custom handling and output formatting of the received data that they desire.

The AMQP Modular Input has such feature.

So you can write your own handler and then declare this in your stanza setup.

Here is an example of some code that you would then compile , jar it up and dump in SPLUNK_HOME/etc/apps/amqp_ta/bin/lib

Then declare the handler to be applied :

Re: How do handle quotes in JSON when receiving AMQP messages

marcusnilssonmr — Mon, 28 Sep 2020 20:18:03 GMT

The pluggable architecture make sense, thanks!. It would be very convenient though, if some more default handlers shipped with the plugin for some obvious choices, such as dumping as is (as you json handler above), and in a format compatible with KV_MODE=auto_escaped.

Re: How do handle quotes in JSON when receiving AMQP messages

marcusnilssonmr — Mon, 15 Jun 2015 21:14:51 GMT

context.stanzaName is not accessible from classes outside of the package, otherwise it compiles fine.

Re: How do handle quotes in JSON when receiving AMQP messages

Damien_Dallimor — Tue, 16 Jun 2015 00:14:21 GMT

Corrected the package name in the example gist.

Re: How do handle quotes in JSON when receiving AMQP messages

Damien_Dallimor — Tue, 16 Jun 2015 00:16:50 GMT

That is one of the core purposes as to why the code repo is on Github , so ideally the community can collaborate and contribute handlers if they see that they are generic enough. Most handlers though will be very proprietary to people's use cases and data, so not warranted to be part of the core release of the AMQP Modular Input.