How can I diff the results of two most recent sources?

[chustar]

I'm currently trying to generate a report describing "what's changed" since the last report.

Currently, my idea is to find the two most recent source files and run a "set diff" on their events.
Unfortunately, while I can find the events from the most recent source file:

| eventstats max(source) as maxsource | where source = maxsource //Our log files are named with increasing numbers.

I can't figure out a way to find the events from the second most recent source file.

[woodcock]

Like this:

... | dedup source | sort 2 - source

Then to get the events from those 2 sources, you do this:

... [search ... | dedup source | sort 2 - source | fields source]

[somesoni2 · Accepted solution]

Try something like this

| set diff [  search 'your base search' [search 'your base search' | dedup source | head 1 | table source] | rest of the search  ]   [search 'your base search' | dedup source | head 2 | reverse | head 1 | table source ]| rest of the search ]

[steveyz]

I'd suggest running a subsearch using either tstats or metadata to get the last 2 sources, and then doing a search like

[<subsearch to get last 2 source names>] | stats dc(source) as dc by _raw | search dc=1 

This will get you all the rows that appear in only 1 of the last 2 sources. Replace '_raw' with whatever other field(s) to fit your needs.

[richgalloway]

Unless your log files are just numbers (not very likely), you'll have to go through some steps to find the previous log. First you'll need to parse maxsource to separate the numeric part from the rest. Then decrement the number and put the two parts back together. Finally, run your set diff using the two sources. Here is an untested example:

... | eventstats max(source) as maxsource | rex field=maxsource "(?P<base>[^\d]+)(?P<numeric>\d+)" | eval numeric=numeric-1 | eval prevsource=base.numeric | set diff [ search source=maxsource] [search source=prevsource] ...

Of course, you'll want to modify the rex command to match your filename format.

[steveyz]

set diff will only work as a generating command (it has to be the first command)

[chustar]

Thanks but the logs arent' incrementing. They numbers increase by arbitrary amounts based on outside factors that can't easily be predicted.