All Apps and Add-ons
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Monitoring of Java Virtual Machines with JMX: Why are we getting "Connection refused to host: 127.0.0.1" errors?

framirez_enova
Explorer
‎04-14-2016 08:36 AM

We are trying to get your JMX app set up to monitor some devices that were moved to a new data-center. They were being monitored from one of our indexers, but now we would be monitoring off a search head as the servers were moved to a new data center. For some reason we keep getting "Connection refused to host: 127.0.0.1" errors.

On the spunk search end we have the following in inputs.conf on the search head:

[jmx://dreamcast]
config_file = configtest4.xml
polling_frequency = 300
sourcetype = jmx
index = jmx
disabled = false

For the devices we are collecting from we have the following in config.xml file on the search head:

<jmxserver jmxServiceURL="service:jmx:rmi:///jndi/rmi://jmxhost.loc.domain.com:1099/jmxrmi" host="jmxhost.loc.domain.com" jvmDescription="dreamcast" jmxport="1099" jmxuser="username" jmxpass="password>

I've read the info on this page: https://splunkbase.splunk.com/app/668/#/documentation , however, I don't see Manager -> DataInputs -> JMX as a place I can work in and am not sure how to proceed.

Of note....

We have both SPLUNK4JMX and jmx_ta installed and after updating them to the most recent versions available we now show two of the same Monitoring of Java Virtual Machines with JMX apps listed.

From the Apps listing this is what I show:

Name                                           Folder name   Version
Monitoring of Java Virtual Machines with JMX   SPLUNK4JMX    2.4
Monitoring of Java Virtual Machines with JMX   jmx_ta      2.1

Here is the output of ./splunk btool outputs list --debug from the jmx host if this helps

/opt/splunkforwarder/etc/system/default/outputs.conf                        [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf                        priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf                        type = udp
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf                        compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        connectionTimeout = 20
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        defaultGroup = domain_indexers_9997
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf                        indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf                        maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf                        readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf                        secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf                        sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf                        sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf                        useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf                        writeTimeout = 300
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        [tcpout:domain_indexers_9997]
/opt/splunkforwarder/etc/apps/EnovaNixForwarder/default/outputs.conf        server = splunkindexer01.dc.domain.com:9997,splunkindexer02.dc.domain.com:9997,splunkindexer03.dc.domain.com:9997

We are not sure how to proceed from here so any help you can provide would be awesome. If you need anything for me please let me know and I can provide it.

Update

I now see this error message as well.... non-JRMP server at remote endpoint

0 Karma
Reply

Damien_Dallimor
Ultra Champion
‎04-19-2016 05:50 PM

You most likely have some sort of firewall/network constraint. I'm guessing here that dreamcast14.loc.domain.com is resolving to 127.0.0.1.

Connection refused to host: 127.0.0.1
java.net.ConnectException: Connection refused
0 Karma
Reply

framirez_enova
Explorer
‎04-20-2016 11:20 AM

We've already ensured with our networking team that there is no blocking on that end.. Firewall is open on the jmx hosts.

0 Karma
Reply

framirez_enova
Explorer
‎04-19-2016 12:18 PM

Should you need the javaversion, this is what I show on the splunksearch server: /usr/lib/jvm/java-7-oracle/jre/bin/java

0 Karma
Reply

Damien_Dallimor
Ultra Champion
‎04-19-2016 09:24 AM

SPLUNK4JMX (community supported) and jmx_ta(splunk supported) , these are 2 entirely different Apps.

Please just install 1 otherwise I can't really make sense of where your logging output is coming from and if your error messages are even JMX related.

With the SPLUNK4JMX app , the correct way to view the log files for debugging is with the Splunk search :
index_internal error ExecProcessor jmx.py

0 Karma
Reply

framirez_enova
Explorer
‎04-25-2016 08:03 AM

I was reading this: https://answers.splunk.com/answers/62185/where-to-install-splunk-for-jmx-app-in-a-distributed-splunk...

It seems we have to deploy the app on the server that the forwarder is setup as well? But that is not stated in the app documentation: https://splunkbase.splunk.com/app/668/#/documentation

Can you please elaborate on that a bit? From the reading I've done it seems we have things setup properly on the Splunk side and have networking allowances in place. The only thing I can see is that on the forwarder there is not a SPLUNK4JMX app installed.

0 Karma
Reply

framirez_enova
Explorer
‎04-21-2016 12:03 PM

Not sure if this helps, but I ran tcpdump -i eth107 -nnn host 00.00.00.00 and port 1099 from dreamcast01.loc.domain.com (the jmx server) to the splunksearch02.loc.domain.com and see the following:

13:13:26.299948 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [S], seq 1540665743, win 29200, options [mss 1460,sackOK,TS val 3664785512 ecr 0,nop,wscale 7], length 0
13:13:26.300336 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 1, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 0
13:13:26.300362 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 1:8, ack 1, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 7
13:13:26.300995 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 0
13:13:26.301085 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 8:25, ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 17
13:13:26.301218 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 25:75, ack 19, win 229, options [nop,nop,TS val 3664785512 ecr 466171144], length 50
13:13:26.305359 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [P.], seq 75:90, ack 246, win 237, options [nop,nop,TS val 3664785513 ecr 466171144], length 15
13:13:56.303081 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [F.], seq 90, ack 246, win 237, options [nop,nop,TS val 3664793013 ecr 466171155], length 0
13:13:56.303453 IP 00.00.00.00.56756 > 00.00.00.00.1099: Flags [.], ack 247, win 237, options [nop,nop,TS val 3664793013 ecr 466178645], length 0
0 Karma
Reply

framirez_enova
Explorer
‎04-19-2016 11:44 AM

Something interesting to note, if this helps any... I looked at the jmx_ta app landing_page and used the look for errors search index=_internal component=ExecProcessor jmx.py and it returns tons of SPLUNK4JMX errors:

04-19-2016 12:52:46.326 -0500 INFO  ExecProcessor - Removing status item "/opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py (isModInput=yes)

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast11.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast11. loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast14.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast14.loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py"    java.net.ConnectException: Connection refused"

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd


04-19-2016 12:48:08.894 -0500 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SPLUNK4JMX/bin/jmx.py" host=dreamcast13.loc.domain.com, jmxServiceURL=service:jmx:rmi:///jndi/rmi://dreamcast13.loc.domain.com:1099/jmxrmi, jmxport=1099, jvmDescription=dreamcast, processID=0,stanza=jmx://dreamcast,systemErrorMessage="Connection refused to host: 127.0.0.1; nested exception is: 

    host = splunksearch02.loc.domain.com
    source = /opt/splunk/var/log/splunk/splunkd.log
    sourcetype = splunkd

FYI, I have disabled the jmx_ta app.

0 Karma
Reply

framirez_enova
Explorer
‎04-19-2016 11:42 AM

Also, using your search with the = (index=_internal error ExecProcessor jmx.py) I get the following:

10.70.70.20 - admin [19/Apr/2016:13:38:29.842 -0500] "GET /en-US/splunkd/_raw/services/search/shelper?output_mode=json&snippet=true&snippetEmbedJS=false&namespace=search&search=search+index_internal+error+ExecProcessor+jmx.py&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&=1461091101858 HTTP/1.1" 200 5747 "https://splunk.domain.com/en-US/app/search/search" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0" - 4fa215f5c097cf4155f80508fa1272bd 99ms

0 Karma
Reply

framirez_enova
Explorer
‎04-19-2016 10:59 AM

Did those get combined? When looking at our setup they go to the same app page ... https://splunkbase.splunk.com/app/668/ When I reached out to the Splunk support team they said it was community based support that I needed.

I ran index_internal error ExecProcessor jmx.py search but nothing comes up so I guess it's running on the jmx_ta setup.

I'm guessing this would need to go to the regular Splunk support portal again?

0 Karma
Reply
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...