PDF Report Server stopped working after 4.3 Upgrade

[tewner]

Hey all!

Our PDF report server stopped working just after our upgrade to 4.3. Any ideas?

python.log

2012-01-19 15:10:09,999 DEBUG Preloading from '/opt/splunk/var/run/splunk/merged/server.conf'.

2012-01-19 15:10:10,000 DEBUG Preloading from '/opt/splunk/var/run/splunk/merged/web.conf'.

2012-01-19 15:10:10,000 DEBUG simpleRequest > GET https://127.0.0.1:8089/servicesNS/nobody/search/admin/alert_actions/email [] sessionSource=direct

2012-01-19 15:10:10,009 DEBUG simpleRequest < server responded status=200 responseTime=0.0082s

2012-01-19 15:10:10,011 INFO sendemail opening PDF request to appserver at http://10.1.34.193:8000/en-US/report/

2012-01-19 15:10:11,165 INFO    pdfhandler:646 - Starting PDF App Renderer Version 1.3

2012-01-19 15:10:11,165 INFO Starting PDF App Renderer Version 1.3

2012-01-19 15:10:11,211 WARNING pdfhandler:458 - body=Splunk;4.3;4.3

2012-01-19 15:10:11,211 WARNING body=Splunk;4.3;4.3

2012-01-19 15:10:11,211 WARNING pdfhandler:459 - entries=['Splunk', '4.3', '4.3']

2012-01-19 15:10:11,211 WARNING entries=['Splunk', '4.3', '4.3']

2012-01-19 15:10:11,227 INFO    xvfb:45 - Started new X server. Now 1 active

2012-01-19 15:10:11,227 INFO Started new X server. Now 1 active

2012-01-19 15:10:11,227 WARNING pdfhandler:551 - Restricting Firefox to following hosts only: *:53 10.1.34.193

2012-01-19 15:10:11,227 WARNING Restricting Firefox to following hosts only: *:53 10.1.34.193

2012-01-19 15:10:11,228 INFO    pdfhandler:558 - Executing /opt/splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox -print http://10.1.34.193:8000/app/search/@go?sid=scheduler__admin__search__MikeTest_at_1326985800_ec4bcd74... -printmode pdf -printfile /tmp/tmpK7Jrfr -title Splunk Alert: MikeTest -orientation portrait -paperwidth 216 -paperheight 279 -mode splunk -timeout 180 -windowsize 782x768 -footer_right Generated by Splunk at &D -cookie session_id_8000=cf12e0248caba3e922699baf368f8373c6201b2a

2012-01-19 15:10:11,228 INFO Executing /opt/splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox -print http://10.1.34.193:8000/app/search/@go?sid=scheduler__admin__search__MikeTest_at_1326985800_ec4bcd74... -printmode pdf -printfile /tmp/tmpK7Jrfr -title Splunk Alert: MikeTest -orientation portrait -paperwidth 216 -paperheight 279 -mode splunk -timeout 180 -windowsize 782x768 -footer_right Generated by Splunk at &D -cookie session_id_8000=cf12e0248caba3e922699baf368f8373c6201b2a

2012-01-19 15:10:11,246 INFO    xvfb:205 - Assigned DISPLAY :5

2012-01-19 15:10:11,246 INFO Assigned DISPLAY :5

2012-01-19 15:10:11,247 INFO    xvfb:115 - Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '3000x768x24', '-nolisten', 'tcp']

2012-01-19 15:10:11,247 INFO Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '3000x768x24', '-nolisten', 'tcp']

2012-01-19 15:10:11,247 INFO    xvfb:116 - Starting X Server env: {'XAUTHORITY': '/opt/splunk/var/run/splunk/xvfb/xauth-5/Xauthority'}

2012-01-19 15:10:11,247 INFO Starting X Server env: {'XAUTHORITY': '/opt/splunk/var/run/splunk/xvfb/xauth-5/Xauthority'}

2012-01-19 15:10:13,253 INFO    xvfb:122 - X Started

2012-01-19 15:10:13,253 INFO X Started

2012-01-19 15:10:40,028 ERROR An error occurred while generating a PDF of this report: Failed to generate PDF: Appserver failed to dispatch report request to /services/pdfserver/renderpdf: Splunkd daemon is not responding: ('The read operation timed out',)

2012-01-19 15:10:40,028 DEBUG simpleRequest > GET https://127.0.0.1:8089/services/search/jobs/scheduler__admin__search__MikeTest_at_1326985800_ec4bcd7... [] sessionSource=direct

2012-01-19 15:10:40,037 DEBUG simpleRequest < server responded status=200 responseTime=0.0092s

2012-01-19 15:10:40,038 DEBUG getStatus - elapsed=0.00961112976074 nextRetry=0.0500000071025

2012-01-19 15:10:40,080 INFO Sending email. subject="Splunk Alert: MikeTest", results_link="http://10.1.34.193:8000/app/search/@go?sid=scheduler__admin__search__MikeTest_at_1326985800_ec4bcd74...", recepients="['mike@XXXXXXX.com']"

2012-01-19 15:10:41,259 INFO    pdfhandler:628 - Generated pdf file.  bytes=332328 time=28.00

2012-01-19 15:10:41,259 INFO Generated pdf file.  bytes=332328 time=28.00

2012-01-19 15:10:41,270 WARNING xvfb:132 - Stopping X Server 5

2012-01-19 15:10:41,270 WARNING Stopping X Server 5

web_service.log :

2012-01-19 15:10:10,026 WARNING [4f18325203220b5290] report:61 - CERT = None

2012-01-19 15:10:10,026 INFO    [4f18325203220b5290] report:190 - Appserver dispatching report request to '/services/pdfserver/renderpdf'

2012-01-19 15:10:28,476 INFO    [4f183264552200e9d0] cached:77 - memoized decorator used on function  with non hashable arguments

2012-01-19 15:10:28,897 INFO    [4f183264552200e9d0] cached:77 - memoized decorator used on function  with non hashable arguments

2012-01-19 15:10:31,954 INFO    [4f183267f321ebbfd0] view:544 - loading SID "scheduler_adminsearchMikeTest_at_1326985800_ec4bcd744f02f7fe"

2012-01-19 15:10:31,981 INFO    [4f183267f321ebbfd0] view:581 - search job was run by saved search scheduler, predefined view: flashtimeline

2012-01-19 15:10:31,987 INFO    [4f183267fc2197af10] cached:77 - memoized decorator used on function  with non hashable arguments

2012-01-19 15:10:32,581 INFO    [4f183267fc2197af10] view:1056 - PERF - viewTime=0.4043s templateTime=0.1898s

2012-01-19 15:10:33,984 INFO    [4f183269fb2aaaac418310] utility:63 - name=javascript, class=Splunk.Session, userAgent=Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20111005 Firefox/4.0.1, platform=Linux x86_64, appCodeName=Mozilla, appName=Netscape, appVersion=5.0 (X11), language=en-US, oscpu=Linux x86_64, product=Gecko, productSub=20111005, buildID=20111005120338, top=0, height=768, width=3000, left=0, pixelDepth=24, colorDepth=24, availWidth=3000, availHeight=768, availLeft=0, availTop=0, documentURL=http://10.1.34.193:8000/en-US/app/search/flashtimeline?media=print&sid=scheduleradminsearchMikeTest_at_1326985800_ec4bcd744f02f7fe, documentReferrer=, flash=10.0.42, Splunk.Session.START_EVENT fired @Thu Jan 19 2012 15:10:33 GMT+0000 (UTC)

2012-01-19 15:10:34,400 INFO    [4f18326a652aaaacc4d9d0] utility:63 - name=javascript, class=Splunk.Session, language=en-US, product=Gecko, appVersion=5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7, platform=Win32, vendor=Google Inc., appCodeName=Mozilla, appName=Netscape, productSub=20030107, userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.75 Safari/535.7, availTop=0, pixelDepth=32, availHeight=1040, height=1080, width=1920, colorDepth=32, availWidth=1920, availLeft=0, documentURL=http://10.1.34.193:8000/en-US/manager/search/saved/searches?msgid=1210670.30214115279&ns=search&redi..., documentReferrer=http://10.1.34.193:8000/en-US/manager/search/saved/searches/MikeTest?action=edit&ns=search&f_ns=sear..., flash=11.1.102, Splunk.Session.START_EVENT fired @Thu Jan 19 2012 17:10:30 GMT+0200 (Jerusalem Standard Time)

2012-01-19 15:10:40,025 ERROR   [4f18325203220b5290] report:193 - Appserver failed to dispatch report request to /services/pdfserver/renderpdf: Splunkd daemon is not responding: ('The read operation timed out',)

2012-01-19 15:10:40,025 ERROR   [4f18325203220b5290] report:18 - SimpleError status=500 message=Appserver failed to dispatch report request to /services/pdfserver/renderpdf: Splunkd daemon is not responding: ('The read operation timed out',)

2012-01-19 15:10:40,025 WARNING [4f18325203220b5290] report:217 - Tearing down session cf12e0248caba3e922699baf368f8373c6201b2a

2012-01-19 15:10:53,502 INFO    [4f18327d7f220a9450] view:544 - loading SID "scheduleradminsearch_MikeTest_at_1326985800_ec4bcd744f02f7fe"

2012-01-19 15:10:53,530 INFO    [4f18327d7f220a9450] view:581 - search job was run by saved search scheduler, predefined view: flashtimeline

2012-01-19 15:10:53,737 INFO    [4f18327dbb2aaaada92ad0] cached:77 - memoized decorator used on function  with non hashable arguments

2012-01-19 15:10:54,295 INFO    [4f18327dbb2aaaada92ad0] view:1056 - PERF - viewTime=0.3912s templateTime=0.1671s

[letienne]

same here. I'd really like to see a solution to this one 🙂

Some reports work, some don't.

[mwagstaff]

Same problem here, following our 4.3 upgrade. Did you manage to get anywhere troubleshooting this one? If not, I'm going to some digging and raise a support case if necessary.

[tewner]

Nope. After a reboot, it started working again, but has since stopped working...