All Apps and Add-ons
Highlighted

How do handle quotes in JSON when receiving AMQP messages

Path Finder

I am using Splunk AMQP Messaging Input with RabbitMQ. When posting a message with JSON content, I receive the following event in Splunk:

Fri Feb 20 11:57:56 CET 2015 name="amqpmsgreceived" eventid="5" msgqueue="splunkconsume" msgexchange="x" msg_body="{"x": 5, "y": 5}"

The problem is that since the json contains ", Splunk interprets msg_body as being "{", since the quotes are not escaped. What is the best way to handle this?

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Ultra Champion

Probably simplest to just do a custom field extraction in Splunk Web.
Follow the "Extract New Fields" link.

alt text

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Path Finder

Can we get the modular input to escape properly? Then all that is needed is a kvmodr that handles escaping.

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Ultra Champion

So when I write Modular Inputs that can conceivably receive any type of payload once they are being used in the wild , I have to employ an architecture that ships with a default message handler (the format you see above) but also allows the end user to plugin their own message handler to perform whatever custom handling and output formatting of the received data that they desire.

The AMQP Modular Input has such feature.

So you can write your own handler and then declare this in your stanza setup.

Here is an example of some code that you would then compile , jar it up and dump in SPLUNKHOME/etc/apps/amqpta/bin/lib

Then declare the handler to be applied :

alt text

View solution in original post

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Path Finder

The pluggable architecture make sense, thanks!. It would be very convenient though, if some more default handlers shipped with the plugin for some obvious choices, such as dumping as is (as you json handler above), and in a format compatible with KVMODE=autoescaped.

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Ultra Champion

That is one of the core purposes as to why the code repo is on Github , so ideally the community can collaborate and contribute handlers if they see that they are generic enough. Most handlers though will be very proprietary to people's use cases and data, so not warranted to be part of the core release of the AMQP Modular Input.

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Path Finder

context.stanzaName is not accessible from classes outside of the package, otherwise it compiles fine.

0 Karma
Highlighted

Re: How do handle quotes in JSON when receiving AMQP messages

Ultra Champion

Corrected the package name in the example gist.

0 Karma