Compare object position with last position with search command

[gccyuen]

I tried to write a search command to track object position, and compare the current position from last position, the date are like:

Obj, x, y
A, 1.1, 2.2
B, 1.1, 2.0
A, 1.2, 2.0
B, 1.3, 2.0
...

So each record of obj 'A' need to compare with the last record of 'A', except for the first record.

However, I found that Splunk push the search result in bunches to the search command, and I will have gaps in between records.

For example when a plain search return 3,148 events, I wrote a pipe search command and find the events arrived in bunches of 50, 449, 2500, 149, 0, 0, 0, the code of the pipe search command is listed below.

import sys

from splunklib.searchcommands import \
    dispatch, StreamingCommand, Configuration, Option, validators

@Configuration()

class CountMatchesCommand(StreamingCommand):

    def stream(self, records):
        i = 0
        for record in records:
            i += 1
            yield record
        self.logger.error('CountMatchesCommand error: %d' % i)

dispatch(CountMatchesCommand, sys.argv, sys.stdin, sys.stdout, __name__)

Can anyone advise what is the best solution to handle the 'bunch arrival' problem so I can reliably compare position with last one?

[aweitzman · Accepted solution]

I'm not sure I understand what your "bunch arrival" problem is. You're just trying to compare the position of each Obj to its previous position, right?

Does this get you what you want?

...your search here...
| streamstats window=2 range(x) as dx range(y) as dy by Obj 
| eval distance=sqrt(pow(dx,2)+pow(dy,2))
| table Obj x y distance

Using streamstats with window=2 will ensure that the range function only compares the current value with the previous one, and results in the difference between the two, which you can then use to compute the distance between the positions.

[gccyuen]

Thanks for your reply, I will take deeper look into the streamstats.
However, I have to check if the object has crossed some line, therefore I have to do it with custom search command. Then I found the custom search command seemed to be invoke multiple times, each time ( bunch) process only a subset of all the events (perhaps for running the search in parallel??). Between two bunches I had a gap and cannot reliably compare the current position with the last position....

[aweitzman]

What do you mean "check if the object has crossed some line"? I'd be very surprised if whatever you need to do requires a custom search command. (That's not to say that it might not be more efficient to use one, but it still might be possible to do it with a regular search.)

[gccyuen]

OK, every minutes I got positions of thousands of objects that are moving around, and I have an arbitrary straight line drawn on the floor, now I need to count the events of objects crossing that line.

So if for an object, its last position is on the 'A' side of the line and the current position is on the 'B' side of the line, then it has just crossed the line, right? The calculation is a bit complex but I did figure it out, and put the algorithm in a custom search command.

The problem now is at search time I cannot reliably get the last position when I have the current event at hand, because Splunk seemed to pass the events in bunches.

[aweitzman]

You can carry the previous value of each event with you using streamstats also:

...your search...
| streamstats window=2 first(x) as prevx first(y) as prevy
| ...your custom command...

Now you don't have to worry about discontinuities in the result batches - the values you need are in the event for you already.

[gccyuen]

I finally work out the solution and your information is critically for solving it!