Where should I check for python.log error messages about generating pdf of scheduled reports?

[skender27]

Hi,

I get the following error while receiving some scheduled reports in Splunk 6.2.

An error occurred while generating the PDF. Please see python.log for details.

But, NOT all reports generate the pdf, only some of them do.
And the following are the lines of the python.log under opt/splunk//var/log/splunk:

2016-01-01 09:01:41,038 +0100 ERROR     pdfgen_endpoint:230 - No views prepared without exceptions. Bailing out of Integrated PDF Generation.
2016-01-01 09:01:41,172 +0100 ERROR     sendemail:971 - An error occurred while generating a PDF: Failed to fetch PDF (status = 400): Unable to render PDF.&lt;br/&gt;<ul><li>Exception raised while trying to prepare "MY OTHER REPORT" for rendering to PDF. Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/MY_APP/search/parser: The read operation timed out',)</li></ul>
2016-01-01 09:01:57,389 +0100 ERROR     __init__:477 - Socket error communicating with splunkd (error=The read operation timed out), path = /servicesNS/admin/servicedesk_plus/search/parser
2016-01-01 09:01:57,390 +0100 ERROR     pdfgen_endpoint:224 - Exception raised while trying to prepare "MY REPORT" for rendering to PDF. Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/MY APP/search/parser: The read operation timed out',)
2016-01-01 09:01:57,390 +0100 ERROR     pdfgen_endpoint:230 - No views prepared without exceptions. Bailing out of Integrated PDF Generation.
2016-01-01 09:01:57,514 +0100 ERROR     sendemail:971 - An error occurred while generating a PDF: Failed to fetch PDF (status = 400): Unable to render PDF.&lt;br/&gt;<ul><li>Exception raised while trying to prepare "MY REPORT" for rendering to PDF. Splunkd daemon is not responding: ('Error connecting to /servicesNS/admin/MY APP/search/parser: The read operation timed out',)</li></ul>

What is exactly the issue?

Thanks for any help,
Skender

[ppablo]

Hi @skender27

Just wanted to follow up to see if this answer by @ronogle worked for you? If yes, please be sure to resolve the post by clicking "Accept" directly below the answer. This is a pretty useful topic that would benefit the community. Thanks!

Patrick

[ronogle · Accepted solution]

I had the same issue. I finally figured it out. To find out if you're having the same issue look splunkd_access.log in $SPLUNK_HOME/var/log/splunk.

user@server:/opt/splunk/var/log/splunk# grep Cisco splunkd_access.log
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.142 -0400] "POST /servicesNS/nobody/search/saved/searches/Audit%20Cisco%20Changes/notify?trigger.condition_state=1 HTTP/1.0" 200 1928 - - - 10ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.339 -0400] "GET /servicesNS/user/search/saved/searches/Audit%20Cisco%20Changes?output_mode=json HTTP/1.0" 200 34334 - - - 15ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.730 -0400] "GET /servicesNS/user/search/saved/searches/Audit%20Cisco%20Changes HTTP/1.0" 200 57101 - - - 11ms
127.0.0.1 - splunk-system-user [17/May/2016:15:26:34.498 -0400] "GET /services/pdfgen/render?paper-size=legal-landscape&sid=scheduler_cm9uLm9nbGU__search__RMD5ec6c9fa239e4c78b_at_1463513100_76&input-report=Audit%20Cisco%20Changes&now=1463513100&owner=user&namespace=search HTTP/1.0" **400** 272 - - - **60452ms**

In the log above, you can see the last line has a "400" status code and a time value of "60452ms". The status code will show up in the python.log.

user@server:/opt/splunk/var/log/splunk# grep Cisco python.log
2016-05-17 15:26:34.498 -0400 ERROR sendemail:996 - An error occurred while generating a PDF: Failed to fetch PDF (**status = 400**): Unable to render PDF.<br/><ul><li>Exception raised while trying to prepare "Audit Cisco Changes" for rendering to PDF. Splunkd daemon is not responding: ("Error connecting to /servicesNS/user/search/search/parser: ('The read operation timed out',)",)</li></ul>
2016-05-17 15:26:34,979 -0400 INFO  sendemail:112 - Sending email. subject="Splunk Report: Audit Cisco Changes", results_link="https://server.admin.local:8000/app/search/@go?sid=scheduler_cm9uLm9nbGU__search__RMD5ec6c9fa239e4c78b_at_1463462400_63889", recipients="[u'hosted@service-now.com']", server="smtp.admin.local"

The error will also show up in the pdfgen.log:

user@server:/opt/splunk/var/log/splunk# grep -i cisco pdfgen.log
2016-05-17 15:26:34,885 -**0400** ERROR pdfgen_utils:481 - Exception raised while trying to prepare "Audit PCI Cisco Changes New" for rendering to PDF. Splunkd daemon is not responding: ("Error connecting to /servicesNS/user/search/search/parser: ('The read operation timed out',)",)

So that is how to verify that you have the same problem that I had. To fix this problem, you have to change the splunkdConnectionTimeout value in web.conf to something bigger than the time found in the splunkd_access.log file. In my case, I changed the value to 90 (seconds). The default value is 30 seconds.

To make the change, copy or create the web.conf file from etc/system/default/web.conf to etc/system/local/web.conf. Change the splunkdConnectionTimeout= where is greater than the number is seconds from the splunkd_access.log. Note that the splunkd_access.log value is in milliseconds.

Restart the Splunkd service.

[the_wolverine]

Very informative writeup. Thank you for taking the time. I'm awarding you points for your effort!

[kwanx]

great write up; thx

[Navanitha]

Well explained. Thank you..

[skender27]

Could it be because of some restart of splunk daemon?

Skender