Splunk Search

How can I diff the results of two most recent sources?

chustar
Path Finder

I'm currently trying to generate a report describing "what's changed" since the last report.

Currently, my idea is to find the two most recent source files and run a "set diff" on their events.
Unfortunately, while I can find the events from the most recent source file:

| eventstats max(source) as maxsource | where source = maxsource //Our log files are named with increasing numbers.

I can't figure out a way to find the events from the second most recent source file.

0 Karma
1 Solution

somesoni2
Revered Legend

Try something like this

| set diff [  search 'your base search' [search 'your base search' | dedup source | head 1 | table source] | rest of the search  ]   [search 'your base search' | dedup source | head 2 | reverse | head 1 | table source ]| rest of the search ]

View solution in original post

woodcock
Esteemed Legend

Like this:

... | dedup source | sort 2 - source

Then to get the events from those 2 sources, you do this:

... [search ... | dedup source | sort 2 - source | fields source]
0 Karma

somesoni2
Revered Legend

Try something like this

| set diff [  search 'your base search' [search 'your base search' | dedup source | head 1 | table source] | rest of the search  ]   [search 'your base search' | dedup source | head 2 | reverse | head 1 | table source ]| rest of the search ]

steveyz
Splunk Employee
Splunk Employee

I'd suggest running a subsearch using either tstats or metadata to get the last 2 sources, and then doing a search like

[<subsearch to get last 2 source names>] | stats dc(source) as dc by _raw | search dc=1 

This will get you all the rows that appear in only 1 of the last 2 sources. Replace '_raw' with whatever other field(s) to fit your needs.

richgalloway
SplunkTrust
SplunkTrust

Unless your log files are just numbers (not very likely), you'll have to go through some steps to find the previous log. First you'll need to parse maxsource to separate the numeric part from the rest. Then decrement the number and put the two parts back together. Finally, run your set diff using the two sources. Here is an untested example:

... | eventstats max(source) as maxsource | rex field=maxsource "(?P<base>[^\d]+)(?P<numeric>\d+)" | eval numeric=numeric-1 | eval prevsource=base.numeric | set diff [ search source=maxsource] [search source=prevsource] ...

Of course, you'll want to modify the rex command to match your filename format.

---
If this reply helps you, Karma would be appreciated.

steveyz
Splunk Employee
Splunk Employee

set diff will only work as a generating command (it has to be the first command)

0 Karma

chustar
Path Finder

Thanks but the logs arent' incrementing. They numbers increase by arbitrary amounts based on outside factors that can't easily be predicted.

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...