when i bounce splunk service, it says splunkd exited (That's what i meant the error messages) which is quite normal i guess? I have just spun up two VMs in aws (One running Rabbit and the other Splunk) and it worked like a charm in the first attempt! Although, i don't see data if i don't provide Exchange name.Bottom line is it works everywhere except at my work. Answers to your questions: I do not see any process on my work machine (one in question) I had a custom script which was grabbing all the messages (Which i have killed and haven't started). So i do not have any other processes running. My Rabbit MQ admin saw a connection only when my custom py script was grabbing messages...Nothing from Splunk (Connectivity exists...literally they are in same subnet and all the ports required have a communication) What's more confusing is, i do not see any error messages....absolutely none!!!! Anything else you suggest me to check? Thanks, Raghav ... View more

Hey Damien, Like i mentioned, i have switched it to the new host (logs i have provided earlier are from a different machine). I have setup a new VM that sits in the same subnet as our RabbitMQ server. Answer to your question is ....There's no firewall or connectivity issues on the new host as well as the old host. I do not see any amqp errors whatsoever on the new host (except when i restart splunkd) my jdk location /opt/jdk1.8 (JAVA_HOME is set to this path) Splunk and AMQP are running fine (no error) I do not see any messages in splunk When i run the custom python script , i get all the messages. PLease let me know if you need more information. Thanks, Raghav ... View more

Thank you Damien, i finally got a new host in the same subnet as the RabbitMQ as this is a POC (it was easy :)) I have a python script that can retrieve messages from that particular queue using pika library. I was able to receive/retrieve messages using the script but i still do not see any messages in Splunk. Also, i do not see any errors now. index=_internal error "amqp.py" (realtime 30 second window) and 0 of 700 events matched. Here's my inputs stanza [amqp://rabbit2] ack_messages = 1 basic_qos_limit = 20 hec_batch_mode = 0 hec_https = 0 hostname = 184.x.x.x index = rabbit index_message_envelope = 1 index_message_propertys = 1 output_type = stdout password = nms port = 5672 queue_name = nms sourcetype = amqp use_ssl = 0 username = nms virtual_host = / Is Exchange name a mandate? I know the config says it picks default i not mentioned. Reason i ask is, i had setup something on my personal laptop and did the same thing...once i punched in the exchange name, messages started to flow in. Also, when i run the custom python script i see a connection established on 5672 but not through amqp app. So it's not credentials or access thing...i do not see a outbound connection being established using the app. Thanks, Raghav ... View more

Hey Damien, Here's today's data 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" ... 12 more 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.HttpService.send(HttpService.java:367) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.Socket.connect(Socket.java:589) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.PlainSocketImpl.socketConnect(Native Method) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" Caused by: java.net.ConnectException: Connection refused 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.modinput.amqp.AMQPModularInput.main(Unknown Source) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.modinput.ModularInput.init(Unknown Source) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.modinput.ModularInput.runStateCheckerThread(Unknown Source) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Entity.getString(Entity.java:279) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Entity.getContent(Entity.java:157) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Entity.validate(Entity.java:462) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Resource.validate(Resource.java:174) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Entity.refresh(Entity.java:1) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Entity.refresh(Entity.java:381) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.HttpService.get(HttpService.java:151) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.Service.send(Service.java:1268) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.HttpService.send(HttpService.java:370) 12-05-2016 22:31:04.526 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" Error executing modular input : Connection refused : java.lang.RuntimeException: Connection refused 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" ... 12 more 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.HttpService.send(HttpService.java:367) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) 12-05-2016 11:16:08.976 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.Socket.connect(Socket.java:589) Thanks, Raghav ... View more

Hey Damien Here's the data from today x.x.x.x - admin [05/Dec/2016:22:25:49.730 -0500] "GET /en-US/splunkd/__raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+error+amqp.py%7Ctable+_&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1480994660560 HTTP/1.1" 200 5209 "http://x.x.x.x:8000/en-US/app/search/search?q=search%20index%3D_internal%20error%20amqp.py&display.page.search.mode=fast&earliest=%40d&latest=now&sid=1480994744.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" - 075eb25b0cfcc1a02f44df93045aebf2 3ms x.x.x.x - admin [05/Dec/2016:22:25:49.056 -0500] "GET /en-US/splunkd/__raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+error+amqp.py%7Ctable+&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1480994660559 HTTP/1.1" 200 5206 "http://x.x.x.x:8000/en-US/app/search/search?q=search%20index%3D_internal%20error%20amqp.py&display.page.search.mode=fast&earliest=%40d&latest=now&sid=1480994744.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" - 075eb25b0cfcc1a02f44df93045aebf2 3ms x.x.x.x - admin [05/Dec/2016:22:25:48.409 -0500] "GET /en-US/splunkd/__raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+error+amqp.py%7Ctab&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1480994660558 HTTP/1.1" 200 5125 "http://x.x.x.x:8000/en-US/app/search/search?q=search%20index%3D_internal%20error%20amqp.py&display.page.search.mode=fast&earliest=%40d&latest=now&sid=1480994744.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" - 075eb25b0cfcc1a02f44df93045aebf2 3ms x.x.x.x - admin [05/Dec/2016:22:25:47.341 -0500] "GET /en-US/splunkd/__raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+error+amqp.py%7C&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1480994660557 HTTP/1.1" 200 6468 "http://x.x.x.x:8000/en-US/app/search/search?q=search%20index%3D_internal%20error%20amqp.py&display.page.search.mode=fast&earliest=%40d&latest=now&sid=1480994744.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" - 075eb25b0cfcc1a02f44df93045aebf2 42ms x.x.x.x - admin [05/Dec/2016:22:24:29.099 -0500] "GET /en-US/splunkd/__raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index%3D_internal+error+amqp.py&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1480994660536 HTTP/1.1" 200 5365 "http://10.63.60.131:8000/en-US/app/search/search" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36" - 075eb25b0cfcc1a02f44df93045aebf2 5ms 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" ... 12 more 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at com.splunk.HttpService.send(HttpService.java:367) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) 12-05-2016 22:05:39.185 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/amqp_ta/bin/amqp.py" at java.net.Socket.connect(Socket.java:589) Thanks, Raghav ... View more