Hi,
After an upgrade from 6.3 to 6.6.1 I am experiencing very slow startups (around 5 minutes) on all search heads, indexers, heavy forwarder, license master and deployment server, while 'Waiting for web server at http://127.0.0.1:8000 to be available' . This also seems to be the case on a standalone dev instance.
I have tracked the issue down to splunk_instrumentation. See the large gap of time in web_service.log without debug logging:
2017-06-06 13:27:04,519 INFO [59369f954b7fb8451b27d0] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 13:31:19,367 INFO [59369f954b7fb8451b27d0] custom:211 - Registering custom app endpoint: splunk_instrumentation/instrumentation_controller
2017-06-06 13:31:19,454 INFO [59369f954b7fb8451b27d0] custom:211 - Registering custom app endpoint: Splunk_TA_nix/setup
And with python debug logging on:
2017-06-06 13:40:04,899 INFO [5936a2a4407fa13c5cd7d0] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 13:40:05,001 DEBUG [5936a2a4407fa13c5cd7d0] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.>
2017-06-06 13:40:05,002 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/instrumentation_controller.py
2017-06-06 13:40:05,002 DEBUG [5936a2a4407fa13c5cd7d0] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.splunk_instrumentation>
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/cherrypy.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/traceback.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/traceback.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/json.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/json.py
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/logging.py'
2017-06-06 13:40:05,004 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/logging.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/zipfile.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/zipfile.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/StringIO.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/StringIO.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/datetime.py'
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/datetime.py
2017-06-06 13:40:05,005 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/cherrypy.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk_instrumentation.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/splunk_instrumentation.py
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/string.py'
2017-06-06 13:40:05,007 DEBUG [5936a2a4407fa13c5cd7d0] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/splunk_instrumentation/appserver/controllers/string.py
2017-06-06 13:44:19,655 INFO [5936a2a4407fa13c5cd7d0] custom:211 - Registering custom app endpoint: splunk_instrumentation/instrumentation_controller
Moving the app folder to somewhere else and starting splunk results in an near instant startup:
2017-06-06 14:29:02,563 INFO [5936ae1c147f2297de8790] root:267 - Proxied mode ip_address=127.0.0.1 port=8065 exposed_port=8000:
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/json.py'
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/json.py
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/setup.py
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.Splunk_TA_nix>
2017-06-06 14:29:02,904 DEBUG [5936ae1c147f2297de8790] custom:148 - LOAD: returning wrapper package: <virtual package: splunk.appserver.mrsparkle.custom_controllers.>
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/splunk.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/splunk.py
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/cherrypy.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/cherrypy.py
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:162 - [Errno 2] No such file or directory: '/opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/sys.py'
2017-06-06 14:29:02,905 DEBUG [5936ae1c147f2297de8790] custom:154 - LOAD: reading file: /opt/splunk/etc/apps/Splunk_TA_nix/appserver/controllers/sys.py
2017-06-06 14:29:02,906 DEBUG [5936ae1c147f2297de8790] root:452 - No shared storage location configured, using: /opt/splunk/var/run/splunk/rss
2017-06-06 14:29:02,906 DEBUG [5936ae1c147f2297de8790] root:446 - Checking for shared storage location
2017-06-06 14:29:02,906 INFO [5936ae1c147f2297de8790] custom:211 - Registering custom app endpoint: Splunk_TA_nix/setup
I have managed to workaround the issue for now by creating /opt/splunk/etc/apps/splunk_instrumentation/local/app.conf with the following parameters:
[install]
allows_disable = true
state = disabled
Hopefully this helps anyone else with slow startup issues, and perhaps someone can shed some light as to why this app, which is supposed to do background telemetry, is causing such an impact.
Thanks,
Carl
... View more