topic Re: Weird behavior with the pow()-function in Splunk Search

Weird behavior with the pow()-function

dkoops — Mon, 07 Mar 2016 09:36:55 GMT

Basically what goes wrong is that the pow() function seems to act weird when exceeding anything above the power 23.
This is the example function and its output:

| eval value = pow(10,22)

This returns 10000000000000000000000.000000, which is what I want.

Next,

| eval value = pow(10,23)

Returns 99999999999999991611392.000000, which is just plain wrong.. Any ideas?

====================================================================

Some more info on why I want to do this, since maybe anyone has a more elegant solution:
My data contains a binary string, say 10001000. I need to join this to a lookup file containing binary masks, so I have to split

10001000 into 10000000 and 1000. The way I do this now is use

| eval masklength1=len(binary)
| eval binary = ltrim(binary,"1")
| eval binary = ltrim(binary,"0")

| eval masklength2=len(binary)
| eval binary = ltrim(binary,"1")
| eval binary = ltrim(binary,"0")

| eval mask1 = pow(10, masklength1-1)
| eval mask2 = pow(10, masklength2-1)

Which seems to do the trick, however when the binary string exceeds 23 characters, Splunk messes it up. I also don't really have an alternative solution to solve this problem.

Re: Weird behavior with the pow()-function

javiergn — Mon, 07 Mar 2016 10:26:07 GMT

I think the problem is that you are trying to work with huge numbers not supported internally by Splunk.
Why don't you try with just strings? I understand you just want to be able to use a lookup after all.

For example, I've written the following for mask1 (you can apply the same logic for 2):

| stats count
| eval binary = "1111111111111111111111111111111111111111111111111111"
| eval masklength1=len(binary)

| eval binary = ltrim(binary,"1")
| eval binary = ltrim(binary,"0")
| eval masklength2=len(binary)

| eval mask1 = mvrange(0, masklength1-1)
| mvexpand mask1
| eval mask1 = if(mask1==0, "1", "0")
| stats list(mask1) as mask1
| eval mask1 = mvjoin(mask1,"")

Output:
100000000000000000000000000000000000000000000000000

Let me know if that helps.

Re: Weird behavior with the pow()-function

dkoops — Mon, 07 Mar 2016 13:13:07 GMT

Good point, leaving it a string. In my case I don't get a single binary string but a large list that I want to join multiple masks on. So to prevent the last "|stats" from joining everything together I did the following:

| eval mask1 = mvrange(0,length1)
| eval mask2 = mvrange(0,length2)
...
| streamstats count
| mvexpand mask1
| mvexpand mask2
...
| eval mask1 = if(mask1==0, "1", "0")
| eval mask2 = if(mask2==0, "1", "0")
...
| stats list(mask1) as mask1 list(mask2) as mask2 ... by count
| eval mask1 = mvjoin(mask1,"")
| eval mask2 = mvjoin(mask2,"")
...

(I need a total of 6 masks to cover the largest binary string I found so far..)

This, however, is impossibly slow..

Re: Weird behavior with the pow()-function

javiergn — Mon, 07 Mar 2016 13:20:02 GMT

If you post your whole query we might be able to help with the optimisation (if there's any possible one :D)

Re: Weird behavior with the pow()-function

dkoops — Mon, 07 Mar 2016 13:38:20 GMT

The whole query is below. With a span that returns less than 10 events it's still quite fast but going over 20 events just keeps is hanging at 'Finalizing Job'.

Another thing I just noticed that goes wrong here is in the case of a binary value such as 11000, the ltrim part goes wrong.. However I think it can be fixed with adding a 'substr(X,Y,Z)'.

I guess if this isn't going to work out I'll have the lookup files changed to a format more easy to use.

    index=abc binary!=0* earliest=-60m
    | eval len1=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval len2=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval len3=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval len4=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval len5=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval len6=len(binary)
    | eval binary = ltrim(binary,"1")
    | eval binary = ltrim(binary,"0")
    | eval mask1 = mvrange(0,len1)
    | eval mask2 = mvrange(0,len2)
    | eval mask3 = mvrange(0,len3)
    | eval mask4 = mvrange(0,len4)
    | eval mask5 = mvrange(0,len5)
    | eval mask6 = mvrange(0,len6)
    | streamstats count
    | mvexpand mask1
    | mvexpand mask2
    | mvexpand mask3
    | mvexpand mask4
    | mvexpand mask5
    | mvexpand mask6
    | eval mask1 = if(mask1==0, "1", "0")
    | eval mask2 = if(mask1==2, "1", "0")
    | eval mask3 = if(mask1==3, "1", "0")
    | eval mask4 = if(mask1==4, "1", "0")
    | eval mask5 = if(mask1==5, "1", "0")
    | eval mask6 = if(mask1==6, "1", "0")
    | stats list(mask1) as mask1 list(mask2) as mask2 list(mask3) as mask3 list(mask4) as mask4 list(mask5) as mask5 list(mask6) as mask6 by count _time
    | eval mask1 = mvjoin(mask1,"")
    | eval mask2 = mvjoin(mask2,"")
    | eval mask3 = mvjoin(mask3,"")
    | eval mask4 = mvjoin(mask4,"")
    | eval mask5 = mvjoin(mask5,"")
    | eval mask6 = mvjoin(mask6,"")
    | dedup count

Re: Weird behavior with the pow()-function

javiergn — Mon, 07 Mar 2016 14:17:11 GMT

That mvexpand is growing the number of events exponentially.
You should also try to prefilter your fields at earliest as possible in your search.

Would the following maybe work for you instead?

index=abc binary!=0* earliest=-60m
| fields binary, _time

| eval len1=len(binary)
| eval binary = if(len1 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval len2=len(binary)
| eval binary = if(len2 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval len3=len(binary)
| eval binary = if(len3 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval len4=len(binary)
| eval binary = if(len4 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval len5=len(binary)
| eval binary = if(len5 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval len6=len(binary)
| eval binary = if(len6 == 0, "0", ltrim(ltrim(binary,"1"),"0"))

| eval mask1 = mvrange(0,len1)
| eval mask2 = mvrange(0,len2)
| eval mask3 = mvrange(0,len3)
| eval mask4 = mvrange(0,len4)
| eval mask5 = mvrange(0,len5)
| eval mask6 = mvrange(0,len6)
| streamstats count

| mvexpand mask1
| eval mask1 = if(mask1==0, "1", "0")
| stats 
    list(mask1) as mask1, 
    values(mask2) as mask2,
    values(mask3) as mask3, 
    values(mask4) as mask4,
    values(mask5) as mask5, 
    values(mask6) as mask6,
    by count, _time
| eval mask1 = mvjoin(mask1,"")

| mvexpand mask2
| eval mask2 = if(mask2==0, "1", "0")
| stats 
    list(mask2) as mask2, 
    values(mask1) as mask1,
    values(mask3) as mask3, 
    values(mask4) as mask4,
    values(mask5) as mask5, 
    values(mask6) as mask6,
    by count, _time
| eval mask2 = mvjoin(mask2,"")

| mvexpand mask3
| eval mask3 = if(mask3==0, "1", "0")
| stats 
    list(mask3) as mask3, 
    values(mask1) as mask1,
    values(mask2) as mask2, 
    values(mask4) as mask4,
    values(mask5) as mask5, 
    values(mask6) as mask6,
    by count, _time
| eval mask3 = mvjoin(mask3,"")

| mvexpand mask4
| eval mask4 = if(mask4==0, "1", "0")
| stats 
    list(mask4) as mask4, 
    values(mask1) as mask1,
    values(mask2) as mask2, 
    values(mask3) as mask3,
    values(mask5) as mask5, 
    values(mask6) as mask6,
    by count, _time
| eval mask4 = mvjoin(mask4,"")

| mvexpand mask5
| eval mask5 = if(mask5==0, "1", "0")
| stats 
    list(mask5) as mask5, 
    values(mask1) as mask1,
    values(mask2) as mask2, 
    values(mask3) as mask3,
    values(mask4) as mask4, 
    values(mask6) as mask6,
    by count, _time
| eval mask5 = mvjoin(mask5,"")

| mvexpand mask6
| eval mask6 = if(mask6==0, "1", "0")
| stats 
    list(mask6) as mask6,
    values(mask1) as mask1,
    values(mask2) as mask2,
    values(mask3) as mask3, 
    values(mask4) as mask4,
    values(mask5) as mask5, 
    by count, _time
| eval mask6 = mvjoin(mask6,"")

Re: Weird behavior with the pow()-function

dkoops — Mon, 07 Mar 2016 14:46:44 GMT

That made it indeed a bit faster, I'll leave it at since it's working now.

I made a request to the people providing the lookup tables if they could incorporate an extra field with the length of the masks so I can join on that. Saves 2/3 of your suggested query 🙂

Anyway thanks a lot for your help, much appreciated!