I have a Dashboard that needs more than one hour for a complete generation. This Dashboard is configured for a scheduled PDF delivery via E-Mail. Once per month at the first day, at 2am. At 3am, I receive an E-Mail saying:
An error occurred while generating the PDF. Please see python.log for details.
In python.log the following lines are written:
ERROR __init__:499 - Socket error communicating with splunkd (error=('The read operation timed out',)), path = /services/pdfgen/render ERROR sendemail:1152 - An error occurred while generating a PDF: Failed to fetch PDF (SplunkdConnectionException): Splunkd daemon is not responding: ("Error connecting to /services/pdfgen/render: ('The read operation timed out',)",) INFO sendemail:134 - Sending email. subject="...", results_link="None", recipients="[u'...', u'...']", server="localhost"
I found a workaround using google and this forum:
I changed these two lines in etc/apps/search/bin/sendemail.py:
PDF_REPORT_SERVER_TIMEOUT = 1800 #was 600 PDFGEN_SIMPLE_REQUEST_TIMEOUT = 10800 #was 3600
PDFGENSIMPLEREQUEST_TIMEOUT is now 3 hours, instead of 1 hour. Now, the dashboard is sent as a PDF via E-Mail correctly. This solution works, but there are 2 problems:
Splunk shows a warning: "Installed Files Integrity Checker: File Integrity checks found 1 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem."
I think i will lose my changes to
etc/apps/search/bin/sendemail.py on Splunk-updates, as it is a file distributed with Splunk and a source-code file, but no configuration file.
What is the correct solution for my problem?
In Line 1284 of
etc/apps/search/bin/sendemail.py (splunk-7.2.3-06d57c595b80-Linux-x86_64.tgz), the function "simpleRequest" is called. There the parameter timeout is set to PDFGENSIMPLEREQUESTTIMEOUT. If this parameter would not be set the function "simpleRequest" would use another value. This is defined in `lib/python2.7/site-packages/splunk/rest/init_.py` line 418 and 419.
But, as the code is in 7.2.3, I think it is not possible to change this behaviour with a configuration file. Is this correct? Notice: that the other value defined in
lib/python2.7/site-packages/splunk/rest/__init__.py:419 (SPLUNKD_CONNECTION_TIMEOUT) is also hardcoded.
Try this seetings in limits.conf( create one in local directory, dont overwrite default):
[pdf] render_endpoint_timeout = <seconds> [scheduler] scheduled_view_timeout = <int>[s|m|h|d]
Thank you for your answer! I tried these settings, but it was not successful. Still the same error. Is this path correct?
As you can see the "[pdf] renderendpointtimeout = " limit is only used in etc/system/bin/pdfgen_endpoint.py:518.
cd /opt/splunk sudo grep -Hirn render_endpoint_timeout . ./etc/system/README/limits.conf.spec:2412:render_endpoint_timeout = <unsigned int> ./etc/system/bin/pdfgen_endpoint.py:518: self._timeoutDuration = int(settings.get('render_endpoint_timeout', self._timeoutDuration)) Übereinstimmungen in Binärdatei ./etc/system/bin/pdfgen_endpoint.pyc. ./etc/system/default/limits.conf:930:render_endpoint_timeout = 3600 ./etc/system/local/limits.conf:13:render_endpoint_timeout = 10800 ./lib/python2.7/site-packages/slim/config/conf-specs/limits.conf.spec:2212:render_endpoint_timeout = <unsigned int>
As i understand the python code in etc/apps/search/bin/sendemail.py it is not possible to change the value of PDFGENSIMPLEREQUEST_TIMEOUT by a config parameter.
Edit: I restarted the splunk server with /opt/splunk/bin/splunk restart after the configuration changes.
Yup, you will lose your changes after upgrading Splunk. Easiest approach here would be to include a note in your internal documentation regarding what changes to test with the new version before upgrading your production splunk.
For a fancy solution that's also upgrade-safe you'd fork your own sendemail.py into an app packaged with your own alert configuration that either overrides the existing configuration in the search app [see configuration file precedence in the docs] or defines an entirely new alert "send email with mad timeouts" to use for such cases.
The latter approach has a big upside: regular users that define email alerts don't hurt themselves or your environment that badly because they're still protected by the standard timeouts.
Finally to note, you can put in an ER with splunk support to make these timeouts configurable.