We have a dashboard containing 8 charts. These charts are rendered using saved searches. The searches make use of a summary index to get the data.
Summary index creation time: Once a day at 00:45 AM.
Schedule for generating PDF and emailing is: Daily at 6:00 AM.
Page size for PDF report is set as: A3
Number of charts per page of PDF is: 2
One chart on this dashboard has some issues. The chart is used to show last 7 days session count.
The search is:
index=my_summary report=my_report earliest=-7d@d latest=-1d | timechart partial=false span=1d sum(sessions) as TotalSessions sum(downloads) as TotalDownloads
On few days the the chart on PDF doesn't contain complete information. The previous day's data will be missing. Whereas if i open the application and view the dashboard on my browser, i see the chart completely.
Note: Some days the same chart on PDF contains complete information.
What could be the problem? How to debug this issue?
What was the time difference you checked the PDF and compared it with running the search manually on dashboard.
The issue can be summary index not getting filled (summary search can be delayed if there are a lot of saved searches running) and the summary index search is getting queued up and getting executed later than 6 AM.
Have you ever seen messages like: Maximum Concurrent Searches that can run on Splunk has reached?
This can be one cause of the problem.
Looking internally to similar issues, there have been a few bug fixes in the past that resolved issues similar to what you are experiencing with PDF exports and charts therein.
What version of Splunk are you all experiencing this issue on?
Looking at what is out there (as fixes or open issues) I'm not seeing anything specific that matches the behavior you're seeing since 6.3.2 (current version is 6.3.3).
So - I'll suggest that you open up a case with as much documentation/example as possible and support will determine if this is a duplicate to an existing open case, whether it might be resolved in the 6.3.3 release (or when a fix might be released), whether there is a work around or patch, or if this is a new 'feature' you've uncovered that has not been reported as of yet.
Any chance this happens on very specific days? Like every Monday and Tuesday? Or Like every day that has a three in the date (3rd,13th,23rd,30th31st). If so perhaps we can narrow it down a bit.