What's the correct way instead of editing etc/apps/search/bin/sendemail.py?

[dfgrtKJH]

I have a Dashboard that needs more than one hour for a complete generation. This Dashboard is configured for a scheduled PDF delivery via E-Mail. Once per month at the first day, at 2am. At 3am, I receive an E-Mail saying:

An error occurred while generating the PDF. Please see python.log for details.

In python.log the following lines are written:

ERROR     __init__:499 - Socket error communicating with splunkd (error=('The read operation timed out',)), path = /services/pdfgen/render
ERROR     sendemail:1152 - An error occurred while generating a PDF: Failed to fetch PDF (SplunkdConnectionException): Splunkd daemon is not responding: ("Error connecting to /services/pdfgen/render: ('The read operation timed out',)",)
INFO      sendemail:134 - Sending email. subject="...", results_link="None", recipients="[u'...', u'...']", server="localhost"

I found a workaround using google and this forum:

I changed these two lines in etc/apps/search/bin/sendemail.py:

PDF_REPORT_SERVER_TIMEOUT = 1800 #was 600
PDFGEN_SIMPLE_REQUEST_TIMEOUT = 10800 #was 3600

PDFGEN_SIMPLE_REQUEST_TIMEOUT is now 3 hours, instead of 1 hour. Now, the dashboard is sent as a PDF via E-Mail correctly. This solution works, but there are 2 problems:

1. Splunk shows a warning: "Installed Files Integrity Checker: File Integrity checks found 1 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem."

2. I think i will lose my changes to etc/apps/search/bin/sendemail.py on Splunk-updates, as it is a file distributed with Splunk and a source-code file, but no configuration file.

What is the correct solution for my problem?

In Line 1284 of etc/apps/search/bin/sendemail.py (splunk-7.2.3-06d57c595b80-Linux-x86_64.tgz), the function "simpleRequest" is called. There the parameter timeout is set to PDFGEN_SIMPLE_REQUEST_TIMEOUT. If this parameter would not be set the function "simpleRequest" would use another value. This is defined in lib/python2.7/site-packages/splunk/rest/__init__.py line 418 and 419.

But, as the code is in 7.2.3, I think it is not possible to change this behaviour with a configuration file. Is this correct? Notice: that the other value defined in lib/python2.7/site-packages/splunk/rest/__init__.py:419 (SPLUNKD_CONNECTION_TIMEOUT) is also hardcoded.

Greetings
Felix

[martin_mueller · Accepted solution]

Yup, you will lose your changes after upgrading Splunk. Easiest approach here would be to include a note in your internal documentation regarding what changes to test with the new version before upgrading your production splunk.

For a fancy solution that's also upgrade-safe you'd fork your own sendemail.py into an app packaged with your own alert configuration that either overrides the existing configuration in the search app [see configuration file precedence in the docs] or defines an entirely new alert "send email with mad timeouts" to use for such cases.
The latter approach has a big upside: regular users that define email alerts don't hurt themselves or your environment that badly because they're still protected by the standard timeouts.

Finally to note, you can put in an ER with splunk support to make these timeouts configurable.

[dfgrtKJH]

Thank you for your answer! I tried these settings, but it was not successful. Still the same error. Is this path correct?
/opt/splunk/etc/system/local/limits.conf

As you can see the "[pdf] render_endpoint_timeout = " limit is only used in etc/system/bin/pdfgen_endpoint.py:518.

cd /opt/splunk
sudo grep -Hirn render_endpoint_timeout .
./etc/system/README/limits.conf.spec:2412:render_endpoint_timeout = <unsigned int>
./etc/system/bin/pdfgen_endpoint.py:518:            self._timeoutDuration = int(settings.get('render_endpoint_timeout', self._timeoutDuration))
Übereinstimmungen in Binärdatei ./etc/system/bin/pdfgen_endpoint.pyc.
./etc/system/default/limits.conf:930:render_endpoint_timeout = 3600
./etc/system/local/limits.conf:13:render_endpoint_timeout = 10800
./lib/python2.7/site-packages/slim/config/conf-specs/limits.conf.spec:2212:render_endpoint_timeout = <unsigned int>

As i understand the python code in etc/apps/search/bin/sendemail.py it is not possible to change the value of PDFGEN_SIMPLE_REQUEST_TIMEOUT by a config parameter.

Edit: I restarted the splunk server with /opt/splunk/bin/splunk restart after the configuration changes.

[p_gurav]

Try this seetings in limits.conf( create one in local directory, dont overwrite default):

 [pdf]
 render_endpoint_timeout = <seconds>

 [scheduler]
 scheduled_view_timeout = <int>[s|m|h|d]

https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf#.5Bpdf.5D