topic Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors? in All Apps and Add-ons

Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Thu, 14 Apr 2016 15:36:12 GMT

We are trying to get your JMX app set up to monitor some devices that were moved to a new data-center. They were being monitored from one of our indexers, but now we would be monitoring off a search head as the servers were moved to a new data center. For some reason we keep getting "Connection refused to host: 127.0.0.1" errors.

On the spunk search end we have the following in inputs.conf on the search head:

[jmx://dreamcast]
config_file = configtest4.xml
polling_frequency = 300
sourcetype = jmx
index = jmx
disabled = false

For the devices we are collecting from we have the following in config.xml file on the search head:

<jmxserver jmxServiceURL="service:jmx:rmi:///jndi/rmi://jmxhost.loc.domain.com:1099/jmxrmi" host="jmxhost.loc.domain.com" jvmDescription="dreamcast" jmxport="1099" jmxuser="username" jmxpass="password>

I've read the info on this page: https://splunkbase.splunk.com/app/668/#/documentation , however, I don't see Manager -> DataInputs -> JMX as a place I can work in and am not sure how to proceed.

Of note....

We have both SPLUNK4JMX and jmx_ta installed and after updating them to the most recent versions available we now show two of the same Monitoring of Java Virtual Machines with JMX apps listed.

From the Apps listing this is what I show:

Name                                           Folder name   Version
Monitoring of Java Virtual Machines with JMX   SPLUNK4JMX    2.4
Monitoring of Java Virtual Machines with JMX   jmx_ta      2.1

Here is the output of ./splunk btool outputs list --debug from the jmx host if this helps

/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        defaultGroup = domain_indexers_9997
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        [tcpout:domain_indexers_9997]
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        server = splunkindexer01.dc.domain.com:9997,splunkindexer02.dc.domain.com:9997,splunkindexer03.dc.domain.com:9997

We are not sure how to proceed from here so any help you can provide would be awesome. If you need anything for me please let me know and I can provide it.

Update

I now see this error message as well.... non-JRMP server at remote endpoint

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

Damien_Dallimor — Tue, 19 Apr 2016 16:24:00 GMT

SPLUNK4JMX (community supported) and jmx_ta(splunk supported) , these are 2 entirely different Apps.

Please just install 1 otherwise I can't really make sense of where your logging output is coming from and if your error messages are even JMX related.

With the SPLUNK4JMX app , the correct way to view the log files for debugging is with the Splunk search :
index_internal error ExecProcessor jmx.py

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Tue, 19 Apr 2016 17:59:02 GMT

Did those get combined? When looking at our setup they go to the same app page ... https://splunkbase.splunk.com/app/668/ When I reached out to the Splunk support team they said it was community based support that I needed.

I ran index_internal error ExecProcessor jmx.py search but nothing comes up so I guess it's running on the jmx_ta setup.

I'm guessing this would need to go to the regular Splunk support portal again?

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Tue, 29 Sep 2020 09:27:16 GMT

Also, using your search with the = (index=_internal error ExecProcessor jmx.py) I get the following:

10.70.70.20 - admin [19/Apr/2016:13:38:29.842 -0500] "GET /en-US/splunkd/_raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index_internal+error+ExecProcessor+jmx.py&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1461091101858 HTTP/1.1" 200 5747 "https://splunk.domain.com/en-US/app/search/search" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" - 4fa215f5c097cf4155f80508fa1272bd 99ms

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Tue, 29 Sep 2020 09:27:19 GMT

Something interesting to note, if this helps any... I looked at the jmx_ta app landing_page and used the look for errors search index=_internal component=ExecProcessor jmx.py and it returns tons of SPLUNK4JMX errors:

04-19-2016 12:52:46.326 -0500 INFO  ExecProcessor - Removing status item "/opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py (isModInput=yes)

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast11.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast11. loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast14.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast14.loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast13.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast13.loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd

FYI, I have disabled the jmx_ta app.

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Tue, 19 Apr 2016 19:18:55 GMT

Should you need the javaversion, this is what I show on the splunksearch server: /usr/lib/jvm/java-7-oracle/jre/bin/java

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

Damien_Dallimor — Wed, 20 Apr 2016 00:50:56 GMT

You most likely have some sort of firewall/network constraint. I'm guessing here that dreamcast14.loc.domain.com is resolving to 127.0.0.1.

Connection refused to host: 127.0.0.1
java.net.ConnectException: Connection refused

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Wed, 20 Apr 2016 18:20:26 GMT

We've already ensured with our networking team that there is no blocking on that end.. Firewall is open on the jmx hosts.

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Thu, 21 Apr 2016 19:03:14 GMT

Not sure if this helps, but I ran tcpdump -i eth107 -nnn host 00.00.00.00 and port 1099 from dreamcast01.loc.domain.com (the jmx server) to the splunksearch02.loc.domain.com and see the following:

13:13:26.299948 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [S], seq 1540665743, win 29200, options [mss 1460,sackOK,TS val 3664785512 ecr 0,nop,wscale 7], length 0
13:13:26.300336 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 1, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 0
13:13:26.300362 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 1:8, ack 1, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 7
13:13:26.300995 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 0
13:13:26.301085 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 8:25, ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 17
13:13:26.301218 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 25:75, ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 50
13:13:26.305359 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 75:90, ack 246, win 237, options [nop,nop,TS val 3664785513 ecr 466171144], length 15
13:13:56.303081 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [F.], seq 90, ack 246, win 237, options [nop,nop,TS val 3664793013 ecr 466171155], length 0
13:13:56.303453 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 247, win 237, options [nop,nop,TS val 3664793013 ecr 466178645], length 0

Re: Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova — Mon, 25 Apr 2016 15:03:17 GMT

I was reading this: https://answers.splunk.com/answers/62185/where-to-install-splunk-for-jmx-app-in-a-distributed-splunk-env.html

It seems we have to deploy the app on the server that the forwarder is setup as well? But that is not stated in the app documentation: https://splunkbase.splunk.com/app/668/#/documentation

Can you please elaborate on that a bit? From the reading I've done it seems we have things setup properly on the Splunk side and have networking allowances in place. The only thing I can see is that on the forwarder there is not a SPLUNK4JMX app installed.