topic Re: Data not ingesting into splunk from RabbitMQ queue in All Apps and Add-ons

Data not ingesting into splunk from RabbitMQ queue

getmesomedata — Sun, 21 Apr 2013 18:50:00 GMT

I'm having some issues trying to get my data from my RabbitMQ instance into splunk.

I've completed the following steps:
- Enabled the STOMP protocol in my installation of RabbitMQ
- Installed the STOMP app in my spunk instance and setup a data input to listen to my queue (127.0.0.1\topic\testQueue)
- Published some messages onto the queue which results in no data in splunk.

I've checked the list of connections within RabbitMQ and there is a connection from splunk so I know that part has worked. I've checked the splunk internal errors and I can't see anything relating to the STOMP app.

Can you suggest any other logs for me to check or is there anything obvious I've missed out?

Re: Data not ingesting into splunk from RabbitMQ queue

allenta — Mon, 22 Apr 2013 15:20:03 GMT

Hi getmesomedata!

The steps you've followed are perfectly correct. It would be helpful if you can make a quick test in order to check if the issue you're experiencing is related with a strange behaviour we are still researching.

We've detected some kind of event buffering somewhere in between RabbitMQ and Splunk. Due to that 'mysterious' buffering, if you test the STOMP modular input with only a few messages, they arrive at Splunk, but they are never rendered in the UI until the buffer is completely filled. So, please, repeat your test with 100 or more messages (you can use the producer.py script in https://github.com/allenta/splunk-stomp/tree/master/extras/clients if you want). Let us know if that way you are able to see the enqueued messages in the Splunk Search UI.

Thank you for the report!

Re: Data not ingesting into splunk from RabbitMQ queue

allenta — Mon, 22 Apr 2013 15:35:04 GMT

And the 'mysterious' buffering issue is now fixed! Please, upgrade to v0.3 and check if your problem persists.

Thank you!

Re: Data not ingesting into splunk from RabbitMQ queue

getmesomedata — Mon, 22 Apr 2013 20:56:46 GMT

Success, v0.3 works a charm! Thanks

Out of curiosity what was the issue?

Re: Data not ingesting into splunk from RabbitMQ queue

allenta — Tue, 23 Apr 2013 06:36:29 GMT

Great 🙂

The issue was trivial. A forgot flush call in the stream which connects the modular input and Splunk. A beginner's mistake.