Streamstats strange behaviour

[ArsenyKapralov]

Hi

I have the following problem. I have a set of events with field called "amount1". In this field I have a number which changes in some events.
I want to check if in latest event value is similar or different with previous one and calculate difference between them and I'm trying to do this with streamstats:

| streamstats current=t window=2 global=f first(amount1) as new_amount1
| eval d=new_amount1-amount1

But I face the following issue:
In event A (latest one) I have actual value of field "amount1" equal to 487. In event B (previous to A) I have actual value of field "amount1" equal to 489. I expect that in event A I'll have field "new_amount1" equal to 489, but it equals to 487. Same time in event B I have new_amount1=487 (I think from event A).

If I try to use latest() instead of first() I have 487 both in amount1 and new_amount1 in event A and both 489 in event B.

Can you help me to fix this?

[sideview · Accepted solution]

Short Version: Try this.

| reverse
| streamstats current=f window=2 global=f first(amount1) as new_amount1
| eval d=new_amount1-amount1

Long Version:

There are some unintuitive things you have to keep in mind about streamstats. Some of these I think you already know, so bear with me.

1) Streamstats goes through the data starting from the most recent event, back to the earliest event. Sometimes when you need it to go from earliest to latest, you need to explicitly stick in a | reverse before the pipe to streamstats. Sometimes you then need another | reverse after. =/

2) first() means the first value that streamstats sees as it works through the rows, which will of course (see point #1) be the latest, and last() will be the earliest. No matter how many years you've known this, it still hurts.

3) Sometimes when what you need is really for streamstats to work the other way (see point #1), you'll try plugging in earliest() and latest() but it wont help and can add to confusion.

4) With current=t first() gets even more confusing, because on a given event if foo is non-null, then first(foo) will always be equal to foo. Unless you're using streamstats to do some fancy surgical null-filling behavior, you probably want to set current=f when you're using first().

5) Beware when using using window=N, and a "by foo" clause on the end of streamstats, that the "global" keyword still defaults to True. So the windowing is calculated globally across all the values of foo.
This is pretty obscure but it's gotten me several times. In such cases you need to remember to set global to false.

So here I think #1 is getting you a little bit, and #4 a little too.

Try this.

| reverse
| streamstats current=f window=2 global=f first(amount1) as new_amount1
| eval d=new_amount1-amount1