search.log in the dispatch directory has this error
12-05-2012 05:17:27.748 ERROR SummaryIndexProcessor - Error moving file '/var/groupon/splunk/var/run/splunk/1354684647.1.tmp' to '/var/groupon/splunk/var/spool/splunk/RMD59f64bf9adfa139f1_1502598233.stash_new'.
As a result the data doesn't arrive in the summary index. Also interesting that the scheduled search is considered successful
What does this error mean? There is plenty of free disk space.
The error means that the results file /var/groupon/splunk/var/run/splunk/1354684647.1.tmp could not be moved into the spool directory /var/groupon/splunk/var/spool/splunk which is where it would be picked up to be indexed, which has a sinkhole (delete after indexing) input set up.
Double check the permissions of the folder /var/groupon/splunk/var/spool/splunk. Is it on the same volume as the rest of Splunk or a different one? Are there any files in /var/groupon/splunk/var/spool/splunk?