Website Monitoring: Why does the web_ping.py process crash after a day or 2 and I have to restart?

[Esky73]

The web_ping.py process crashes after a day or 2. Not sure why - a restart resumes it (works fine in our test env - but falls over regularly in prod)

Looked at this article :

https://answers.splunk.com/answers/236703/website-monitoring-why-does-the-app-stop-monitorin.html

See the following in the internal search :

20:20:06,598 ERROR Execution failed
Traceback (most recent call last):
  File "/app/splunk/splunk/etc/apps/website_monitoring/bin/modular_input.py", line 1277, in execute
    self.do_run(in_stream, log_exception_and_continue=True)
  File "/app/splunk/splunk/etc/apps/website_monitoring/bin/modular_input.py", line 1177, in do_run
    input_config)
  File "/app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py", line 462, in run
    proxy_type, proxy_server, proxy_port, proxy_user, proxy_password = self.get_proxy_config(input_config.session_key, conf_stanza)
  File "/app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py", line 430, in get_proxy_config
    website_monitoring_config = WebsiteMonitoringConfig.get( WebsiteMonitoringConfig.build_id( stanza, "website_monitoring", "nobody"), sessionKey=session_key )
  File "/app/splunk/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 548, in get
    return SplunkRESTManager(cls, sessionKey=sessionKey).get(id)
  File "/app/splunk/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 528, in get
    entity = self._get_entity(id, host_path=host_path)
  File "/app/splunk/splunk/lib/python2.7/site-packages/splunk/models/base.py", line 444, in _get_entity
    return self._fix_entity(splunk.entity.getEntity(self.model.resource, None, sessionKey=self.sessionKey, uri=id))
  File "/app/splunk/splunk/lib/python2.7/site-packages/splunk/entity.py", line 249, in getEntity
    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
  File "/app/splunk/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 478, in simpleRequest
    raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /servicesNS/nobody/website_monitoring/admin/website_monitoring/default: The read operation timed out',)

[LukeMurphey]

This should no longer be an issue with version 1.6.1 and later. The input was modified such that it tries again later to get the proxy information and thus gracefully recovers.

[Esky73]

Hi any insight into this issue ?

thanks

[Esky73]

Fell over again .. a bit more info :

05-27-2016 20:18:35.032 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py" InsecureRequestWarning) 
05-27-2016 20:19:24.918 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized 
05-27-2016 20:19:26.083 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py" /app/splunk/splunk/etc/apps/website_monitoring/bin/website_monitoring_app/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html 
05-27-2016 20:19:26.083 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py" InsecureRequestWarning) 
05-27-2016 20:20:06.600 +1000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "error". 
05-27-2016 20:20:06.608 +1000 WARN ExecProcessor - Streaming XML data: Expected tag "event", instead received "message". 
05-27-2016 20:20:06.684 +1000 WARN TcpOutputProc - Pipeline data does not have indexKey. [_path] = /app/splunk/splunk/etc/apps/website_monitoring/bin/web_ping.py\n[_raw] = \n[_meta] = punct::\n[_stmid] = uKuLYMRZJ5Ga5pC\n[MetaData:Source] = source::web_ping\n[MetaData:Host] = host::HOSTNAME\n[MetaData:Sourcetype] = sourcetype::web_ping\n[_done] = _done\n[_linebreaker] = _linebreaker\n[_conf] = source::web_ping|host::HOSTNAME|web_ping|0\n[_channel] = 0\n 
05-27-2016 20:20:06.820 +1000 WARN HttpListener - Socket error from 127.0.0.1 while accessing /servicesNS/nobody/website_monitoring/admin/website_monitoring/default: Broken pipe 
05-27-2016 20:20:21.634 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized 
05-27-2016 20:20:46.812 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized 
05-27-2016 20:21:01.426 +1000 INFO TcpOutputProc - Connected to idx=xxx.xxx.xxx.xxx:9997 using ACK. 
05-27-2016 20:21:11.991 +1000 ERROR ExecProcessor - message from "python /app/splunk/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 401 Client Error: Unauthorized