It would be helpful to see what settings you are using with your summary search. How long does the search usually take to run? Did you set a schedule window for the search? Is this the only summary search you have that is displaying this behavior?
There are a couple of places to look, if you haven't already.
Check the spool directory ($SPLUNK_HOME/var/spool/splunk) on the server you are executing the summary search (I assume its a search head). This is where the summarized data sits before it is forwarded to the indexers. If you have files backing up here, that could indicate a bottleneck with moving the data over. If that is the case, check how much data the SH is forwarding, and then check the pipelines on the indexer (the DMC has good dashboards for this).
Check the _internal log for errors associated with your summary search (index=_internal sourcetype=scheduler). Look for the status field of the search (success, continued, skipped, etc). You can also search for ( index=_internal sourcetype=splunkd (searchscheduler OR dispatchmanager)) that should show you any errors related to scheduled searches, and you can see the searchscheduler metrics (group=searchscheduler). You might be hitting either a per-user or per-system search limit, and that would cause erratic behavior in your summary search performance.
... View more
