Splunk Search

How to get stats command to calculate with precision for numbers in scientific notation?

dmr195
Communicator

I have found that the stats command's output doesn't use scientific notation. This means that if I need to calculate some statistics on a set of very small numbers then stats just reports 0.00000.

You can see the effect with this search:

| stats count | eval small="1.23456e-306" | stats min(small)

It will always return 0.00000.

Is there a way to have the stats command in the above search keep the precision and return 1.23456e-306?

(I know in this example the stats command is pointless because there's only one value, but the real case is that I have many events containing very small numbers and I want to calculate various statistics of the data set without all the small numbers getting changed to 0.)

Tags (1)
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try something like this

| stats count | eval small="1.23456e-30" | eval small=small*1 | stats min(small)

PS: The example value that you gave little too small so changed to little big number.

View solution in original post

dmr195
Communicator

A possibility for getting around the loss of precision in stats is to take logarithms before the stats command and then convert back afterwards. This works quite well:

| stats count | eval small="1.23456e-304" | eval lnsmall=ln(small) | stats min(lnsmall) as minlnsmall | eval minsmall=if(minlnsmall<-702.0, 0, exp(minlnsmall*1.0))

The check for minlnsmall being less than -702 is needed to avoid nan being returned close to the limits of double precision floating point accuracy, and it's necessary to multiply minlnsmall by 1.0 rather than 1 otherwise some premature rounding occurs sometimes (but not always).

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try something like this

| stats count | eval small="1.23456e-30" | eval small=small*1 | stats min(small)

PS: The example value that you gave little too small so changed to little big number.

dmr195
Communicator

Another interesting thing is that eval can handle the full double precision accuracy, for example:

| stats count | eval small="1.23456e-303" | eval small=small*1

prints small with 308 digits to the right of the decimal point.

So it seems like it's the stats command that's limited to 125 decimal places.

dmr195
Communicator

It seems like 125 digits to the right of the decimal point is the most that's possible. So:

| stats count | eval small="1.23456e-125" | eval small=small*1 | stats min(small)

goes to:

0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001

and:

| stats count | eval small="1.23456e-126" | eval small=small*1 | stats min(small)

loses all precision and prints 0.

This is a shame as the smallest double precision floating point number is 2.2e-308. But with I'm better off than I was originally, so thanks for the answer!

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...