topic Re: How to use "set diff" and "diff" commands to list out new values in a file comparing different days/weeks/months/years? in Splunk Search

How to use "set diff" and "diff" commands to list out new values in a file comparing different days/weeks/months/years?

chandana204 — Wed, 23 May 2018 16:47:48 GMT

I have data which add new files every day. I want to compare today's data with previous day/week/month/year data and list out new files. Is it possible to list out? I have gone through Splunk answers to find solutions. I found below search to compare files from yesterday.

sourcetype=nessus source=*Host_Enumeration* earliest=-1d@d latest=now NOT [search sourcetype=nessus source=*Host_Enumeration* earliest=-3d@d latest=-2d@d | stats count by dest_ip | dest_ip]| stats count by dest_ip | dest_ip

I changed the time-range in the search to find new files from previous week/month/year as below

sourcetype=nessus source=*Host_Enumeration* earliest=-1d@d latest=now NOT [search sourcetype=nessus source=*Host_Enumeration* earliest=-6w@d latest=-2d@d | stats count by dest_ip | dest_ip]| stats count by dest_ip | dest_ip

With the above code, the result has previous day (earliest=-3d@d latest=-2d@d) files.

Why it is listing previous files? Is there any other possibility to find solution for this scenario?

Thanks in Advance
Chandana

Re: How to use "set diff" and "diff" commands to list out new values in a file comparing different days/weeks/months/years?

DalJeanis — Wed, 23 May 2018 17:18:56 GMT

updated to mark code.

Re: How to use "set diff" and "diff" commands to list out new values in a file comparing different days/weeks/months/years?

DalJeanis — Wed, 23 May 2018 17:24:26 GMT

There is seldom a good use case for set diff, and this is not it.

Let's ask the question a different way: how can we find the date/time of the first occurrence of each file? Well, let's just ask for that, and if the first occurrence is within the last day, 2 days, whatever, then we will mark it as "new".

Given your code, I'm assuming that dest is the name of the file.

    sourcetype=nessus source=Host_Enumeration earliest=-30d
   | fields dest
   | dedup dest
   | stats min(_time) as _time by dest
   | where _time >= relative_time(now(),"-1d@d")

Now, the above will work. For future reference, though, if the information is in the summary statistics, then you really want to be using tstats because it is screaming fast. So, perhaps if we were checking for new sources or sourcetypes, then you'd want to use a tstats method rather than checking laboriously through the individual events.

https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Tstats

Re: How to use "set diff" and "diff" commands to list out new values in a file comparing different days/weeks/months/years?

chandana204 — Wed, 23 May 2018 18:15:10 GMT

Thank you for your response. My main issue in this task is list out new files. here new files means files are not existed in previous data . If i use above query I can't list out new files but i can see all files with timestamp. Unfortunately, this is not the answer what I am looking for.

To understanding more i'll give an example

today: r, t, y, w, h, g, u, o, p, l, i, a, c

yesterday: y, i, c, b, f, j, 1, 9, 5, 3, aa, gf, br, rh
last week: w, g, fg, rd, o, ff, ht, og, c, y
last 6 months: y, hd, jw, o, r, kd, rd
last year: sd, ed, ewe, ui, oo, kas, w, c

Different files in today's data: t, h, u, p, l, a
The rest of the files are occurred in yesterday, last week, last 6 months and last year.

Can i get the result as above using splunk tool?

I already mentioned this point but i'm again mentioning i.e., the search query what I have mentioned in my question, it is working fine but it is listing different files from previous day only. My requirement is, I want compare all files from previous week/month/year too.

Thanks,
Chandana