topic Re: How can I diff the results of two most recent sources? in Splunk Search

How can I diff the results of two most recent sources?

chustar — Tue, 11 Aug 2015 16:40:27 GMT

I'm currently trying to generate a report describing "what's changed" since the last report.

Currently, my idea is to find the two most recent source files and run a "set diff" on their events.
Unfortunately, while I can find the events from the most recent source file:

| eventstats max(source) as maxsource | where source = maxsource //Our log files are named with increasing numbers.

I can't figure out a way to find the events from the second most recent source file.

Re: How can I diff the results of two most recent sources?

richgalloway — Tue, 11 Aug 2015 18:06:20 GMT

Unless your log files are just numbers (not very likely), you'll have to go through some steps to find the previous log. First you'll need to parse maxsource to separate the numeric part from the rest. Then decrement the number and put the two parts back together. Finally, run your set diff using the two sources. Here is an untested example:

... | eventstats max(source) as maxsource | rex field=maxsource "(?P<base>[^\d]+)(?P<numeric>\d+)" | eval numeric=numeric-1 | eval prevsource=base.numeric | set diff [ search source=maxsource] [search source=prevsource] ...

Of course, you'll want to modify the rex command to match your filename format.

Re: How can I diff the results of two most recent sources?

steveyz — Tue, 11 Aug 2015 18:14:19 GMT

set diff will only work as a generating command (it has to be the first command)

Re: How can I diff the results of two most recent sources?

steveyz — Tue, 11 Aug 2015 18:18:15 GMT

I'd suggest running a subsearch using either tstats or metadata to get the last 2 sources, and then doing a search like

[<subsearch to get last 2 source names>] | stats dc(source) as dc by _raw | search dc=1

This will get you all the rows that appear in only 1 of the last 2 sources. Replace '_raw' with whatever other field(s) to fit your needs.

Re: How can I diff the results of two most recent sources?

somesoni2 — Tue, 11 Aug 2015 18:25:22 GMT

Try something like this

| set diff [  search 'your base search' [search 'your base search' | dedup source | head 1 | table source] | rest of the search  ]   [search 'your base search' | dedup source | head 2 | reverse | head 1 | table source ]| rest of the search ]

Re: How can I diff the results of two most recent sources?

woodcock — Tue, 11 Aug 2015 19:03:56 GMT

Like this:

... | dedup source | sort 2 - source

Then to get the events from those 2 sources, you do this:

... [search ... | dedup source | sort 2 - source | fields source]

Re: How can I diff the results of two most recent sources?

chustar — Tue, 11 Aug 2015 22:19:26 GMT

Thanks but the logs arent' incrementing. They numbers increase by arbitrary amounts based on outside factors that can't easily be predicted.