After updating to the latest version of the PDF Server App, the app stopped functioning for us. Could this be a problem with the PDF Server? How can we troubleshoot it?
We are on x86-64 SuSE Linux Enterprise Server, Service Pack 3.
Unfortunately, SuSE Linux Enterprise Server 10 ships a version of libfontconfig (2.3.94) which is a beta version of the up-and-coming libfontconfig 2.4. At least, it was this way around the time that the SuSE distribution upstream from SuSE Enterprise was stabilizing in the spring of 2006. 2.4 was eventually released in September of the same year.
This version of libfontconfig has a bug which was subsequently fixed in source around April 11, 2006 ( http://lists.freedesktop.org/archives/fontconfig/2006-April/002246.html ) and released as libfontconf 2.3.95 without this flaw on April 27, 2006.
So, unfortunately, this problem you are having was fixed 6.5 years ago. Perhaps it is time to upgrade.
You can identify the problem clearly in $SPLUNK_HOME/var/log/splunk/python.log by seeing glibc failure messages of the following nature (dates removed for brevity):
ERROR Execution of Firefox for x86_64 failed
ERROR pdfhandler:625 - FF STDERR: *** glibc detected *** /splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox-bin: realloc(): invalid next size: 0x000000000099da30 ***
ERROR FF STDERR: *** glibc detected *** /splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox-bin: realloc(): invalid next size: 0x000000000099da30 ***
ERROR pdfhandler:625 - FF STDERR: ======= Backtrace: =========
ERROR FF STDERR: ======= Backtrace: =========
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6[0x2b6612cd46de]
ERROR FF STDERR: /lib64/libc.so.6[0x2b6612cd46de]
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6[0x2b6612cd73bd]
ERROR FF STDERR: /lib64/libc.so.6[0x2b6612cd73bd]
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6(__libc_realloc+0x11c)[0x2b6612cd863c]
ERROR FF STDERR: /lib64/libc.so.6(__libc_realloc+0x11c)[0x2b6612cd863c]
ERROR pdfhandler:625 - FF STDERR: /usr/lib64/libfontconfig.so.1(FcObjectToPtr+0x261)[0x2b661128b9c1]
ERROR FF STDERR: /usr/lib64/libfontconfig.so.1(FcObjectToPtr+0x261)[0x2b661128b9c1]
The key thing here is that glibc notices bad memory calls, and the
caller is libfontconfig.so.1
Unfortunately, SuSE Linux Enterprise Server 10 ships a version of libfontconfig (2.3.94) which is a beta version of the up-and-coming libfontconfig 2.4. At least, it was this way around the time that the SuSE distribution upstream from SuSE Enterprise was stabilizing in the spring of 2006. 2.4 was eventually released in September of the same year.
This version of libfontconfig has a bug which was subsequently fixed in source around April 11, 2006 ( http://lists.freedesktop.org/archives/fontconfig/2006-April/002246.html ) and released as libfontconf 2.3.95 without this flaw on April 27, 2006.
So, unfortunately, this problem you are having was fixed 6.5 years ago. Perhaps it is time to upgrade.
You can identify the problem clearly in $SPLUNK_HOME/var/log/splunk/python.log by seeing glibc failure messages of the following nature (dates removed for brevity):
ERROR Execution of Firefox for x86_64 failed
ERROR pdfhandler:625 - FF STDERR: *** glibc detected *** /splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox-bin: realloc(): invalid next size: 0x000000000099da30 ***
ERROR FF STDERR: *** glibc detected *** /splunk/etc/apps/pdfserver/bin/firefox-x86_64/firefox-bin: realloc(): invalid next size: 0x000000000099da30 ***
ERROR pdfhandler:625 - FF STDERR: ======= Backtrace: =========
ERROR FF STDERR: ======= Backtrace: =========
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6[0x2b6612cd46de]
ERROR FF STDERR: /lib64/libc.so.6[0x2b6612cd46de]
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6[0x2b6612cd73bd]
ERROR FF STDERR: /lib64/libc.so.6[0x2b6612cd73bd]
ERROR pdfhandler:625 - FF STDERR: /lib64/libc.so.6(__libc_realloc+0x11c)[0x2b6612cd863c]
ERROR FF STDERR: /lib64/libc.so.6(__libc_realloc+0x11c)[0x2b6612cd863c]
ERROR pdfhandler:625 - FF STDERR: /usr/lib64/libfontconfig.so.1(FcObjectToPtr+0x261)[0x2b661128b9c1]
ERROR FF STDERR: /usr/lib64/libfontconfig.so.1(FcObjectToPtr+0x261)[0x2b661128b9c1]
The key thing here is that glibc notices bad memory calls, and the
caller is libfontconfig.so.1