All Apps and Add-ons

How do handle quotes in JSON when receiving AMQP messages

marcusnilssonmr
Path Finder

I am using Splunk AMQP Messaging Input with RabbitMQ. When posting a message with JSON content, I receive the following event in Splunk:

Fri Feb 20 11:57:56 CET 2015 name="amqp_msg_received" event_id="5" msg_queue="splunk_consume" msg_exchange="x" msg_body="{"x": 5, "y": 5}"

The problem is that since the json contains ", Splunk interprets msg_body as being "{", since the quotes are not escaped. What is the best way to handle this?

0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

So when I write Modular Inputs that can conceivably receive any type of payload once they are being used in the wild , I have to employ an architecture that ships with a default message handler (the format you see above) but also allows the end user to plugin their own message handler to perform whatever custom handling and output formatting of the received data that they desire.

The AMQP Modular Input has such feature.

So you can write your own handler and then declare this in your stanza setup.

Here is an example of some code that you would then compile , jar it up and dump in SPLUNK_HOME/etc/apps/amqp_ta/bin/lib

Then declare the handler to be applied :

alt text

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

So when I write Modular Inputs that can conceivably receive any type of payload once they are being used in the wild , I have to employ an architecture that ships with a default message handler (the format you see above) but also allows the end user to plugin their own message handler to perform whatever custom handling and output formatting of the received data that they desire.

The AMQP Modular Input has such feature.

So you can write your own handler and then declare this in your stanza setup.

Here is an example of some code that you would then compile , jar it up and dump in SPLUNK_HOME/etc/apps/amqp_ta/bin/lib

Then declare the handler to be applied :

alt text

0 Karma

marcusnilssonmr
Path Finder

context.stanzaName is not accessible from classes outside of the package, otherwise it compiles fine.

0 Karma

Damien_Dallimor
Ultra Champion

Corrected the package name in the example gist.

0 Karma

marcusnilssonmr
Path Finder

The pluggable architecture make sense, thanks!. It would be very convenient though, if some more default handlers shipped with the plugin for some obvious choices, such as dumping as is (as you json handler above), and in a format compatible with KV_MODE=auto_escaped.

0 Karma

Damien_Dallimor
Ultra Champion

That is one of the core purposes as to why the code repo is on Github , so ideally the community can collaborate and contribute handlers if they see that they are generic enough. Most handlers though will be very proprietary to people's use cases and data, so not warranted to be part of the core release of the AMQP Modular Input.

0 Karma

Damien_Dallimor
Ultra Champion

Probably simplest to just do a custom field extraction in Splunk Web.
Follow the "Extract New Fields" link.

alt text

0 Karma

marcusnilssonmr
Path Finder

Can we get the modular input to escape properly? Then all that is needed is a kvmodr that handles escaping.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...