topic Re: How to find differences between two searches with "set diff" command? in Splunk Search

How to find differences between two searches with "set diff" command?

evang_26 — Sun, 17 Aug 2014 19:03:42 GMT

Hi,

It's been more than a week that I am trying to display the difference between two search results in one field using the "| set diff" command diff. However, it seems to be impossible and very difficult. Below is my code:

| set diff [search sourcetype=nessus source=*Host_Enumeration* earliest=-3d@d latest=-2d@d | eval day="Yesterday"  | timechart count(dest_ip) by dest_ip] [search sourcetype=nessus source=*Host_Enumeration* earliest=-1d@d latest=now| eval day="Today" |timechart count(dest_ip) by dest_ip]

Now, the output of that should be only one dst IP address, but what I get as a result is a big timechart with all the dst IPs. It seems like the "| set diff" command never applied to it? I don't know.

Can you help me please?

Regards,
Evang

Re: How to find differences between two searches with "set diff" command?

Ayn — Mon, 18 Aug 2014 06:39:51 GMT

The output will not be what you want, because you're comparing two sets that will be completely different.

The first set will have a number of values for _time that correspond to the time periods the first search covers, which is from 3 days ago up until 2 days ago. The second set on the other hand will have times that include the last day up until now. So set diff will look at these sets, compare them and see that these are different on every line.

I'm not sure I follow what you want to do with timechart if all you want is two sets of IP's (also not sure what the eval command before that is for). If you tell us more about what you're trying to achieve it will be easier to help you. set diff would be one option, but this sounds like something that should be doable just using subsearches as well.

Re: How to find differences between two searches with "set diff" command?

evang_26 — Mon, 28 Sep 2020 17:21:16 GMT

Hi Ayn,

Thank you for your answer!

You correctly understanded that I want to compare just two sets of IPs, nothing else.

No specific reason why I used timechart in the end. About eval, nothing special again, I saw it in an example with set diff.

I played also with the following (if that is called subsearch) without any results.

sourcetype=nessus source=Host_Enumeration earliest=-3d@d latest=-2d@d NOT [ search sourcetype=nessus source=Host_Enumeration earliest=-1d@d latest=now] | timechart count(dest_ip) by dest_ip]

Any ideas, just for the IP set comparison?

Regards,
Evangelos

Re: How to find differences between two searches with "set diff" command?

somesoni2 — Mon, 18 Aug 2014 15:55:58 GMT

Try something like this

To get list of dest_ip which are not common between Yesterday and Today.

| set diff [search sourcetype=nessus source=*Host_Enumeration* earliest=-3d@d latest=-2d@d | stats count by dest_ip |table dest_ip] [search sourcetype=nessus source=*Host_Enumeration* earliest=-1d@d latest=now | stats count by dest_ip |table dest_ip]

If you need to know list of dest_ip present Yesterday but not Today.

sourcetype=nessus source=*Host_Enumeration* earliest=-3d@d latest=-2d@d NOT [search sourcetype=nessus source=*Host_Enumeration*  earliest=-1d@d latest=now | stats count by dest_ip | dest_ip]| stats count by dest_ip | dest_ip

If you need to know list of dest_ip present Today but not Yesterday.

sourcetype=nessus source=*Host_Enumeration* earliest=-1d@d latest=now NOT [search sourcetype=nessus source=*Host_Enumeration* earliest=-3d@d latest=-2d@d | stats count by dest_ip | dest_ip]| stats count by dest_ip | dest_ip

Re: How to find differences between two searches with "set diff" command?

evang_26 — Mon, 18 Aug 2014 16:35:55 GMT

Hi somesoni2,

Thank you for your response. I would go with the first option, that is find which IPs are not in common.

However, this doesn't work prompting that "dest" command doesn't exist.

I tried to add the "search" command before dest_ip on each end. That time I saw no error coming, but no results as well.

Last, I tried removing all the last part, making it end at stats count by dest_ip. I got nothing again, though I know that there is a difference.

Hmm..

Regards,
Evangelos

Re: How to find differences between two searches with "set diff" command?

somesoni2 — Mon, 18 Aug 2014 17:11:33 GMT

That's my bad, I missed 'table' command there. Try the updated answer.

Re: How to find differences between two searches with "set diff" command?

evang_26 — Mon, 18 Aug 2014 20:24:40 GMT

Hi somesoni2,

Thank you so much! That worked!

Regards,
Evang

Re: How to find differences between two searches with "set diff" command?

meganarellano — Fri, 06 Oct 2017 08:32:06 GMT

Thank you! After beating my head against a wall for a week I finally found this Answer. I removed the "stats count by" as the results were the same either way but otherwise, it worked like a charm as is.

Re: How to find differences between two searches with "set diff" command?

sarmavempati — Fri, 19 Jan 2018 18:11:24 GMT

Can you suggest how to display the results side by side, rather than one after another?
Output:
A B

1
2
3
4
5

===================

I would rather like it be:

1 4
2 5
3
etc.

Thanks

Re: How to find differences between two searches with "set diff" command?

sarmavempati — Fri, 19 Jan 2018 18:14:01 GMT

Formatting lost!

...A.......................B...
...1........................x
...2........................x
...3.........................x
.............................4

.............................5

Desired:
.....A..............B
.....1..............4
......2.............5
......3.................
etc.

Re: How to find differences between two searches with "set diff" command?

guru89044 — Tue, 20 Feb 2018 06:08:09 GMT

This is not working for me.

index="x" AND host= y AND "java.exception" AND earliest=-1d@d latest=now
NOT [search (index=x) AND (host=y AND "java.exception" AND earliest=-3d@d latest=-2d@d
| rex "(?java?.[.\w]+Exception)" | stats count by Exception | sort by count ] | rex "(?java?.[.\w]+Exception)" | stats count by Exception | sort by count

its not listing today' exceptions. Its listing all the exceptions from today and yesterday and ignoring common ones

Re: How to find differences between two searches with "set diff" command?

smahone11 — Wed, 26 Sep 2018 17:07:30 GMT

This example also just returns all the results from the first subsearch, even though both searches return the exact same results

Re: How to find differences between two searches with "set diff" command?

ephrem3232 — Fri, 12 Jun 2020 02:21:51 GMT

Thank you so much. Took me a whole day to figure it out, but finally did it. Thank you again.