topic Re: Report Dedup in Splunk Search

Report Dedup

TucoRameriz — Thu, 25 Apr 2013 19:00:51 GMT

Is there a way to eliminate duplicates by reports? Specifically what I'm looking to do is run a report every 24hrs for X range | stat count by shost. I don't want the report to show any hosts that showed up on the previous report. I know how to eliminate duplicates from a single report but don't know if it's possible to "dedup" on a previously run report.

Thanks

Re: Report Dedup

martin_mueller — Thu, 25 Apr 2013 21:44:38 GMT

You could do something like this:

your  report with stat count for the current day | search NOT [query for shost values in the previous day]

That will remove rows if the shost value appeared in the previous day... once translated into proper splunk commands of course.

Re: Report Dedup

kristian_kolb — Fri, 26 Apr 2013 06:28:29 GMT

martin_muller is essentially right, the solution below might be slightly more efficient, as the stats is performed on only the smaller set of data.

sourcetype=xxx index=yyy earliest=@d NOT [sourcetype=xxx index=yyy earliest=-1d@d latest=@d | dedup shost | fields + shost] | stats c by shost

The subsearch will execute first and return the distinct set of shost for the previous day, so that the outer search will effectively be;

sourcetype=xxx index=yyy earliest=@d NOT ((shost=host1) OR (shost=host2) OR (shost=host3)) | stats c by shost

Hope this helps,

Kristian