As I said, it depends on what you are putting in your summary index and how you would detect and cancel out duplicates.
In my case, my summary index is a collection of counts over 5 minute periods across a number of dimensions.
Let's say the report covers a 1 hour period counting events by host and service in 5 minute buckets
index=x earliest=-1h@h latest=@h
| bin span=5m _time
| stats count by _time host service
| summaryindex spool=t uselb=t addtime=t index="summary" name="services"
These events can be retrieved from the summary index like so
index=summary search_name=services earliest=-1h@h latest=@h
However, some of the fields in the original report have been renamed and repurposed, e.g. host from the original report is now orig_host and host holds the host name of the search head which put the entries in the summary index.
So, to find duplicates and remove, the renames have to be corrected (so the "comparison" will work) and the aggregations have to be negated (so they can be subtracted from the values returned by the rerunning search). Note that I am only keeping counts that are greater than zero, which fits my usecase, but you might want counts greater than or equal to zero.
index=x earliest=-1h@h latest=@h
| bin span=5m _time
| stats count by _time host service
| append
[ search index=summary search_name=services earliest=-1h@h latest=@h
| eval host=orig_host
| eval sourcetype=orig_sourcetype
| eval index=orig_index
| eval count = -count
| table _time, index, sourcetype, host, service, count]
| stats sum(count) as count by _time, index, sourcetype, host, service
| where count > 0
| summaryindex spool=t uselb=t addtime=t index="summary" name="services"
Caveats:
- Subsearches are usually truncated at 50,000 events, so depending on how many entries you are trying to dedup, you may need to do multiple appends over different time periods.
- Aggregations such as averages may return different results due to floating point processors being slightly different between two runs
- You don't need to specify the summary index command if your search is marked as a summary index search, this is added automatically as can be seen if you view the results of the search.
- As always, answers given are just suggestions, it is your responsibility to ensure that your implementation works for your usecase.
Hopefully, this will give you some clues as to how you might achieve this for your usecase.
