Splunk Search

What does stats partitions do?

ryhluc01
Communicator

What does stats partitions do? How would you use this?

Sample query:

|stats **partitions=1** latest(Insert_Text) by field_1 field_2 

The partition and by fields can be whatever you specify. I just would like to understand what is going on here.

lstewart_splunk
Splunk Employee
Splunk Employee

The partitions argument runs the reduce step (in parallel reduce processing) with multiple threads in the same search process on the same machine. Compare that with parallel reduce that runs the reduce step in parallel on multiple machines.

Parallel reduce is implemented with the redistribute command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Redistribute

In testing, the run times for a search using partitions=5 shows no difference as compared to partitions=1, using this search:

| makeresults count=5000000
| streamstats count as i
| eval a=floor(i/10), b=floor(i/100), c=floor(i/1000), d=floor(i/10000), e=floor(i/100000)
| stats partitions=5 count by a, b, c, d, e

In this example, we are able to observe an ~9 second difference in run times:

| makeresults count=500000
| streamstats count as i
| eval j=mvrange(0,10)
| stats partitions=15 count by i,j

With partitions=15 the search completes in 27 seconds. With partitions=1 the search completes in 36 seconds.

The likely issue with the partitions argument as compared to the redistribute command, is the threads are competing for memory on the same machine - and memory usage is one of the significant factors that cause high-cardinality stats to perform poorly. Partitioning the memory usage across different machines, as the redistribute command does, eliminates that competition.

woodcock
Esteemed Legend

Well the dox say that 1 is the default value for partitions so your search can drop it with no change. The documentation is useless on this parameter. You ask a good question.

ryhluc01
Communicator

Right? The documentation on the subject is not helpful at all. Thanks @woodcock. Now I am just hoping someone else will know.

0 Karma

woodcock
Esteemed Legend

I submitted feedback at the bottom of the documentation page, asking for more detail.

0 Karma

ryhluc01
Communicator

Thanks @woodcock! You're the best.

0 Karma

woodcock
Esteemed Legend

If everybody in any community does his little part, the experience for everybody improves dramatically.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...