How do handle quotes in JSON when receiving AMQP messages

[marcusnilssonmr]

I am using Splunk AMQP Messaging Input with RabbitMQ. When posting a message with JSON content, I receive the following event in Splunk:

Fri Feb 20 11:57:56 CET 2015 name="amqp_msg_received" event_id="5" msg_queue="splunk_consume" msg_exchange="x" msg_body="{"x": 5, "y": 5}"

The problem is that since the json contains ", Splunk interprets msg_body as being "{", since the quotes are not escaped. What is the best way to handle this?

[Damien_Dallimor · Accepted solution]

So when I write Modular Inputs that can conceivably receive any type of payload once they are being used in the wild , I have to employ an architecture that ships with a default message handler (the format you see above) but also allows the end user to plugin their own message handler to perform whatever custom handling and output formatting of the received data that they desire.

The AMQP Modular Input has such feature.

So you can write your own handler and then declare this in your stanza setup.

Here is an example of some code that you would then compile , jar it up and dump in SPLUNK_HOME/etc/apps/amqp_ta/bin/lib

Then declare the handler to be applied :

[marcusnilssonmr]

context.stanzaName is not accessible from classes outside of the package, otherwise it compiles fine.

[Damien_Dallimor]

Corrected the package name in the example gist.

[marcusnilssonmr]

The pluggable architecture make sense, thanks!. It would be very convenient though, if some more default handlers shipped with the plugin for some obvious choices, such as dumping as is (as you json handler above), and in a format compatible with KV_MODE=auto_escaped.

[Damien_Dallimor]

That is one of the core purposes as to why the code repo is on Github , so ideally the community can collaborate and contribute handlers if they see that they are generic enough. Most handlers though will be very proprietary to people's use cases and data, so not warranted to be part of the core release of the AMQP Modular Input.

[Damien_Dallimor]

Probably simplest to just do a custom field extraction in Splunk Web.
Follow the "Extract New Fields" link.

[marcusnilssonmr]

Can we get the modular input to escape properly? Then all that is needed is a kvmodr that handles escaping.