Splunk Search

How to get stats command to calculate with precision for numbers in scientific notation?

Communicator

I have found that the stats command's output doesn't use scientific notation. This means that if I need to calculate some statistics on a set of very small numbers then stats just reports 0.00000.

You can see the effect with this search:

| stats count | eval small="1.23456e-306" | stats min(small)

It will always return 0.00000.

Is there a way to have the stats command in the above search keep the precision and return 1.23456e-306?

(I know in this example the stats command is pointless because there's only one value, but the real case is that I have many events containing very small numbers and I want to calculate various statistics of the data set without all the small numbers getting changed to 0.)

Tags (1)
1 Solution

SplunkTrust
SplunkTrust

Try something like this

| stats count | eval small="1.23456e-30" | eval small=small*1 | stats min(small)

PS: The example value that you gave little too small so changed to little big number.

View solution in original post

Communicator

A possibility for getting around the loss of precision in stats is to take logarithms before the stats command and then convert back afterwards. This works quite well:

| stats count | eval small="1.23456e-304" | eval lnsmall=ln(small) | stats min(lnsmall) as minlnsmall | eval minsmall=if(minlnsmall<-702.0, 0, exp(minlnsmall*1.0))

The check for minlnsmall being less than -702 is needed to avoid nan being returned close to the limits of double precision floating point accuracy, and it's necessary to multiply minlnsmall by 1.0 rather than 1 otherwise some premature rounding occurs sometimes (but not always).

0 Karma

SplunkTrust
SplunkTrust

Try something like this

| stats count | eval small="1.23456e-30" | eval small=small*1 | stats min(small)

PS: The example value that you gave little too small so changed to little big number.

View solution in original post

Communicator

Another interesting thing is that eval can handle the full double precision accuracy, for example:

| stats count | eval small="1.23456e-303" | eval small=small*1

prints small with 308 digits to the right of the decimal point.

So it seems like it's the stats command that's limited to 125 decimal places.

Communicator

It seems like 125 digits to the right of the decimal point is the most that's possible. So:

| stats count | eval small="1.23456e-125" | eval small=small*1 | stats min(small)

goes to:

0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001

and:

| stats count | eval small="1.23456e-126" | eval small=small*1 | stats min(small)

loses all precision and prints 0.

This is a shame as the smallest double precision floating point number is 2.2e-308. But with I'm better off than I was originally, so thanks for the answer!