I'm trying to use acceleration for three reports. The Report Acceleration Summary reports the summary status of these accelerations at 33%, and they never seem to get past that point. If I rebuild, they again get to 33% (and quickly), and then don't update past that status.
Splunk acceleration never completes because it hits a search time limit. In my case I determined that it was set to 600 seconds or 10 minutes. I have been fighting this recently. How did I come to this conclusion? I had a search which collects millions of records and must add up and convert the amounts based on a currency lookup table. I wanted to accelerate this search over a 1 month period. The acceleration got stuck at 44%. By default splunk has the acceleration schedule set to happen every 10 minutes. You can change it by adding this in the savedSearch.conf:
auto_summarize.cron_schedule = 3,13,23,33,43,53 * * * *
In this particular example, I was shifting the schedule so that it does not happen at the 0, 10, 20,30,40,50 minutes because too many accelerations triggered huge load spikes.
Anyway, I changed the schedule so that the acceleration process happens only once an hour. Then I watched it. At the specified time the updating message appeared, and exactly 10 minutes later, on the dot, it showed 44%. Then next message was "pending" which is what happens when the acceleration process has nothing to do. And the same thing happened again at the scheduled time.
So, my recommendation is: if you can, use a smaller time range, or use a different technique, such as building a dataset in the summary index with a query that runs every 5 minutes, and then use your regular search to get the data to show.
This isn't an answer exactly, so I won't frame it as such--but you probably won't get a better idea of what's going on until you investigate the logs of the summarize creation search. It sounds like you have a problem bucket that is causing the summarize creation search to stall or crash. If you're not sure how to perform this analysis, contact Splunk Support.
Update: I was also trying to figure out why other users weren't getting acceleration for these searches like I was. All the permissions were appropriate, but their searches ran slow, and mine didn't. I thought maybe Splunk was using old jobs to return my results more quickly, but testing that out on a different user account didn't work. Likewise, erasing my jobs didn't slow me down on the main account. So I turned acceleration off for the searches, and my user account's searches were slow again.
I just gone into Summry Details for one of the searches and used the verfiy button. Nothing came back, however, when I refresh the Summary Deatils page, a notice pops up that verifcation failed, and then that notification disappears. If i catch before it disappears, the failure details tell me that some buckets failed. Anyway, splunk doc states "you can review the root search string (or strings) to see if it can be fixed to provide correct result". The search runs fine, and if splunk thinks otherwise, it certainly isn't providing any clues as to why.
Also, when I refresh the Report Acceleration Summary, my three searches show the "Building" icon and 34%, then after a few seconds, the icon disappears, and the status goes back down to 33%.