How to edit my search so I can compare differences between a list of fields

[splunker1981]

Hello all,

At a loss trying to accomplish the following:

I would like to compare three fields in the same index (test index with 1 sourcetype) going back 2 hours and find any/all values that were added/removed or updated from 3 fields between the current hour and the previous. I ingest data in once an hour, so in reality it's just checking for changes between current events and the previous hour. For the most part events coming in should be relative the same with maybe a handful of updates in a 24hr period - meaning it's a static list which gets update when transactions occur. Here is the command I was trying to use to accomplish this.

|set diff [search  index=tempTest earliest=-1h@h latest=now|fields location, record, status] 
[ index=search index=tempTest  earliest=-2h@h latest=-1h@h fields location, record, status]  

I however don't get any results although there's definitely a handful of differences. Is there a better way to get a list of differences similar to diff's output. This is another search I tried which got me somewhat closer but it would not detect events where there was a slight update to the event in a given field - for example a digit going from 999883 to 999884 on the record field would not display as a change. I'm also not quite clear on how to show which were added/removed - meaning which were found in the last hour and not in the current hour and vice versa. Any help would be greatly appreciated.

| index  index=tempTest earliest=-1h@h latest=now | fields location, record, status
| eval input="1"
| append 
    [search index=tempTest  earliest=-2h@h latest=-1h@h| eval input="2" | fields location, record, status |eval input="2"]
| stats count by location, record, status, input

Example of changes would be as follows:

current hour

location record status
chicago,us    S    500 --> changed from A to S
chicago.us    T    200
sanFan,us     A    200
dc,us         T    500
uk            X    200  ---> changed from A to X and value from 100 to 200
madrid        X    900  ---> change added, didn't exist previous run

previous hour

chicago,us    A    500 
chicago.us    T    200
sanFan,us     A    200
dc,us         T    500
uk            A    100 

So what I'd like to see is this in the results

previous           current               change_type
chicago-1-500     chicago-S-500      modified 
uk-A-100          uk-X-200           modified
                  madrid-X-90        added

[DalJeanis]

I'm not sure I understand what you mean by finding "events removed from the index".

If something is removed from the index, then a search should not be finding it.

Now, you could output a copy of the events you want to track, and then compare to a copy an hour later...but that's a lot of data.

So, could you give a little more insight into what you are referring to as an event, here?

[splunker1981]

thanks for the reply, made some updates to the question, hopefully that clears things up

[DalJeanis]

The second "eval input="2" - line 5 - is probably not supposed to be there.