Smart-mode search emits only one row of stats but gets auto-finalized for disk usage

[jspears]

I have a user whose monthly report search is being auto-finalized due to disk usage. I've ensured there are no other jobs hanging around, and it's being run in Smart mode. Here's what the search looks like (base search modified for public posting):

index=redacted
| stats latest(time_taken) AS time_taken latest(responseTime) AS responseTime by CorrelationId 
| eval responseTime=(time_taken * 1000) 
| stats count, avg(responseTime) as AvgTime, p99(responseTime) as p99, p95(responseTime) as p95, p90(responseTime) as p90, p50(responseTime) as p50

This is over tens of millions of events, but it produces only one row of output, as expected. My only theory so far is that the first stats output gets saved to dispatch along with the final output, but that seems... wrong.

[martin_mueller · Accepted solution]

Your initial stats triggers a massive on-disk search, exhausting the user's dispatch space. On slack you mentioned you have a tiny duplication issue which made you add that first stats - it's probably not really relevant to your overall results. Try this:

index=redacted
| stats dc(CorrelationId) as count, avg(time_taken) as AvgTime, p99(time_taken) as p99, p95(time_taken) as p95, p90(time_taken) as p90, p50(time_taken) as p50
| foreach AvgTime p* [eval <<FIELD>> = <<FIELD>> * 1000]

Note that I've moved the eval to after the stats, then you don't have to do millions of evals for each single event.

[sloshburch]

Also, it might help to toss in the sourcetype. If there are many sourceytpes in the index then filtering on that in the base search could reduce the payload returned.