topic How to get top 20 results from each category based on data size? in Splunk Search

How to get top 20 results from each category based on data size?

VatsalJagani — Wed, 30 Mar 2022 22:32:16 GMT

I've below search:

| tstats summariesonly=true count, sum(All_Traffic.bytes) as total_bytes, sum(All_Traffic.packets) as total_packets from datamodel=Network_Traffic by All_Traffic.src_ip, All_Traffic.dest_ip, All_Traffic.action | rename "All_Traffic.*" as * | stats sum(total_bytes) as total_bytes, sum(total_packets) as total_packets by src_ip dest_ip action | sort 0 -total_bytes | streamstats count as count by action | search count<=20

The purpose of using the last 3 lines with sort and streamstats is I want the top 20 results by total_bytes from each value of the action field.

The only problem with this solution is that streamstats has a limit of 10000 in limits.conf. Do we have any better solution for this?

Re: Get top 20 results from each category based on data size

PickleRick — Wed, 30 Mar 2022 21:33:50 GMT

"The maxresultrows setting specifies a top limit for the window argument. This sets the number of result rows that the streamstats command processor can store in memory. The max_mem_usage_mb setting limits how much memory the streamstats command uses to keep track of information."

It doesn't mean that it cannot process more than 10000 rows of results.

It means that it can use at most maxresultrows to keep track of the stats. And that's completely different.

For example, for a simple

streamstats count

Splunk needs to keep track only of one counter.

If you count by some field, it needs to keep track of count of each encountered field value and so on.

To be honest, I'm not even sure those get counted against the limit.

If you use a bigger window, Splunk has to remember previous values of results to calculate stats over the window. That's where the limit might kick in.

So if you have relatively few actions, you shouldn't even notice the streamstats impact memorywise.

Re: Get top 20 results from each category based on data size

VatsalJagani — Thu, 31 Mar 2022 05:20:13 GMT

Just has two action values. "allowed" and "blocked", and I still notice this issue.

I'm okay if there is another solution than streamstats to achieve this outcome.