JMX add-on failing once a day at same time, every day: java.net.SocketException: Socket is closed

[michael_sleep]

Been wrestling with this one for a while. Every day at the same time the JMX add-on stops working with the following error in the jmx.log:

2018-01-24 05:27:01,638 - com.splunk.modinput.ModularInput -60118777 [main] ERROR  - Error executing modular input : java.lang.RuntimeException: java.net.SocketException: Socket is closed : java.lang.RuntimeException: java.lang.RuntimeException: java.net.SocketException: Socket is closed
    at com.splunk.Entity.refresh(Unknown Source)
    at com.splunk.Entity.refresh(Unknown Source)
    at com.splunk.Resource.validate(Unknown Source)
    at com.splunk.Entity.validate(Unknown Source)
    at com.splunk.Entity.validate(Unknown Source)
    at com.splunk.Resource.getTitle(Unknown Source)
    at com.splunk.Resource.getName(Unknown Source)
    at com.splunk.jmx.JMXModularInputV3.doRun(Unknown Source)
    at com.splunk.modinput.ModularInput.init(Unknown Source)
    at com.splunk.jmx.JMXModularInputV3.main(Unknown Source)
Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
    at com.splunk.AtomObject.createReader(Unknown Source)
    at com.splunk.AtomFeed.parseStream(Unknown Source)
    ... 10 more
Caused by: javax.xml.stream.XMLStreamException: java.net.SocketException: Socket is closed
    at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.<init>(Unknown Source)
    at com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl(Unknown Source)
    at com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(Unknown Source)
    ... 12 more
Caused by: java.net.SocketException: Socket is closed
    at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)
    at sun.security.ssl.AppInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at sun.net.www.MeteredStream.read(Unknown Source)
    at java.io.FilterInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDocumentEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.setInputSource(Unknown Source)
    ... 16 more

I've tried updating to the latest JMX app. The latest version of the JDK/JRE, etc. Based on the majority of the 'working' log it looks like JMX refreshes it's JMX server information every minute:

2018-01-24 05:20:46,081 - org.exolab.castor.mapping.Mapping -59743220 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:20:46,137 - com.splunk.modinput.ModularInput -59743276 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:20:50,777 - com.splunk.modinput.ModularInput -59747916 [main] INFO   - stanza count:1
2018-01-24 05:21:01,352 - com.splunk.modinput.ModularInput -59758491 [main] INFO   - stanza count:1
2018-01-24 05:21:11,914 - com.splunk.modinput.ModularInput -59769053 [main] INFO   - stanza count:1
2018-01-24 05:21:22,496 - com.splunk.modinput.ModularInput -59779635 [main] INFO   - stanza count:1
2018-01-24 05:21:33,084 - com.splunk.modinput.ModularInput -59790223 [main] INFO   - stanza count:1
2018-01-24 05:21:43,644 - com.splunk.modinput.ModularInput -59800783 [main] INFO   - stanza count:1
2018-01-24 05:21:54,209 - com.splunk.modinput.ModularInput -59811348 [main] INFO   - stanza count:1
2018-01-24 05:21:54,796 - org.exolab.castor.mapping.Mapping -59811935 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:21:54,853 - com.splunk.modinput.ModularInput -59811992 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:22:04,806 - com.splunk.modinput.ModularInput -59821945 [main] INFO   - stanza count:1
2018-01-24 05:22:15,387 - com.splunk.modinput.ModularInput -59832526 [main] INFO   - stanza count:1
2018-01-24 05:22:26,009 - com.splunk.modinput.ModularInput -59843148 [main] INFO   - stanza count:1
2018-01-24 05:22:36,567 - com.splunk.modinput.ModularInput -59853706 [main] INFO   - stanza count:1
2018-01-24 05:22:47,138 - com.splunk.modinput.ModularInput -59864277 [main] INFO   - stanza count:1
2018-01-24 05:22:57,720 - com.splunk.modinput.ModularInput -59874859 [main] INFO   - stanza count:1
2018-01-24 05:23:03,632 - org.exolab.castor.mapping.Mapping -59880771 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:23:03,692 - com.splunk.modinput.ModularInput -59880831 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:23:08,317 - com.splunk.modinput.ModularInput -59885456 [main] INFO   - stanza count:1
2018-01-24 05:23:18,895 - com.splunk.modinput.ModularInput -59896034 [main] INFO   - stanza count:1
2018-01-24 05:23:29,520 - com.splunk.modinput.ModularInput -59906659 [main] INFO   - stanza count:1
2018-01-24 05:23:40,109 - com.splunk.modinput.ModularInput -59917248 [main] INFO   - stanza count:1
2018-01-24 05:23:50,668 - com.splunk.modinput.ModularInput -59927807 [main] INFO   - stanza count:1
2018-01-24 05:24:01,390 - com.splunk.modinput.ModularInput -59938529 [main] INFO   - stanza count:1
2018-01-24 05:24:03,634 - org.exolab.castor.mapping.Mapping -59940773 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:24:03,693 - com.splunk.modinput.ModularInput -59940832 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:24:11,950 - com.splunk.modinput.ModularInput -59949089 [main] INFO   - stanza count:1
2018-01-24 05:24:22,509 - com.splunk.modinput.ModularInput -59959648 [main] INFO   - stanza count:1
2018-01-24 05:24:33,110 - com.splunk.modinput.ModularInput -59970249 [main] INFO   - stanza count:1
2018-01-24 05:24:43,696 - com.splunk.modinput.ModularInput -59980835 [main] INFO   - stanza count:1
2018-01-24 05:24:54,316 - com.splunk.modinput.ModularInput -59991455 [main] INFO   - stanza count:1
2018-01-24 05:25:03,721 - org.exolab.castor.mapping.Mapping -60000860 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:25:03,787 - com.splunk.modinput.ModularInput -60000926 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:25:05,076 - com.splunk.modinput.ModularInput -60002215 [main] INFO   - stanza count:1
2018-01-24 05:25:15,652 - com.splunk.modinput.ModularInput -60012791 [main] INFO   - stanza count:1
2018-01-24 05:25:26,285 - com.splunk.modinput.ModularInput -60023424 [main] INFO   - stanza count:1
2018-01-24 05:25:36,869 - com.splunk.modinput.ModularInput -60034008 [main] INFO   - stanza count:1
2018-01-24 05:25:47,514 - com.splunk.modinput.ModularInput -60044653 [main] INFO   - stanza count:1
2018-01-24 05:25:58,144 - com.splunk.modinput.ModularInput -60055283 [main] INFO   - stanza count:1
2018-01-24 05:26:03,739 - org.exolab.castor.mapping.Mapping -60060878 [Thread-2] INFO   - Loading mapping descriptors from jar:file:/C:/Splunk/etc/apps/Splunk_TA_jmx/bin/lib/jmxmodinput.jar!/mapping.xml
2018-01-24 05:26:03,807 - com.splunk.modinput.ModularInput -60060946 [Thread-2] INFO   - 552 servers found in stanza jmx://websitejava
2018-01-24 05:26:08,929 - com.splunk.modinput.ModularInput -60066068 [main] INFO   - stanza count:1
2018-01-24 05:26:19,499 - com.splunk.modinput.ModularInput -60076638 [main] INFO   - stanza count:1
2018-01-24 05:26:30,047 - com.splunk.modinput.ModularInput -60087186 [main] INFO   - stanza count:1
2018-01-24 05:26:40,639 - com.splunk.modinput.ModularInput -60097778 [main] INFO   - stanza count:1
2018-01-24 05:26:51,213 - com.splunk.modinput.ModularInput -60108352 [main] INFO   - stanza count:1
2018-01-24 05:27:01,638 - com.splunk.modinput.ModularInput -60118777 [main] ERROR  - Error executing modular input : java.lang.RuntimeException: java.net.SocketException: Socket is closed : java.lang.RuntimeException: java.lang.RuntimeException: java.net.SocketException: Socket is closed
    at com.splunk.Entity.refresh(Unknown Source)
    at com.splunk.Entity.refresh(Unknown Source)
    at com.splunk.Resource.validate(Unknown Source)
    at com.splunk.Entity.validate(Unknown Source)
    at com.splunk.Entity.validate(Unknown Source)
    at com.splunk.Resource.getTitle(Unknown Source)
    at com.splunk.Resource.getName(Unknown Source)
    at com.splunk.jmx.JMXModularInputV3.doRun(Unknown Source)
    at com.splunk.modinput.ModularInput.init(Unknown Source)
    at com.splunk.jmx.JMXModularInputV3.main(Unknown Source)
Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
    at com.splunk.AtomObject.createReader(Unknown Source)
    at com.splunk.AtomFeed.parseStream(Unknown Source)
    ... 10 more
Caused by: javax.xml.stream.XMLStreamException: java.net.SocketException: Socket is closed
    at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.<init>(Unknown Source)
    at com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl(Unknown Source)
    at com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(Unknown Source)
    ... 12 more
Caused by: java.net.SocketException: Socket is closed
    at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)
    at sun.security.ssl.AppInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at sun.net.www.MeteredStream.read(Unknown Source)
    at java.io.FilterInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDocumentEntity(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.setInputSource(Unknown Source)
    ... 16 more

but for whatever reason after 24 hours it fails with a "socket closed" and doesn't recover. I'm sure it's a configuration issue somewhere but haven't been sure where to look

[Damien_Dallimor]

Trying using our app , http://splunkbase.splunk.com/app/668/ , which is the original JMX App.

[felipesewaybric]

I was having almost the same problem, after several attempts, the only working solution to me was updating my centos5 to centos7 and bam... no more Errors with the JMX.

[harsmarvania57]

Looks like bug in this add-on, I can see same error in our environment but JMX restarted automatically after "Socket is closed" error and we never need to restart splunk to start JMX again. I would suggest you to raise case with splunk.

[Damien_Dallimor]

You tagged your post with the wrong app, the above logs are not from my “monitoring of jvms with jmx” app.

[michael_sleep]

Updated the tag.