Delete events after stats operation

[bhavesh0124]

I'm running stats to find out which events I want to delete. Basically I'm finding the minimum "change_set" a particular "source" has. Now, I want to delete these sources with the least "change_set".  

Note: the events also has a lot of attributes apart from change_set and source.

index=test 
| stats min(change_set) by source

(Now delete the events which has that source and change_set.

```

How can I write the delete operation with this query? (the most optimal way)

@gcusello @ITWhisperer @scelikok @PickleRick 

Thanks

[ITWhisperer]

Delete is a non-recoverable command so great care must be taken when used it.

To identify the events to delete, try something like this

index=test 
| eventstats min(change_set) as min_change_set by source
| where change_set = min_change_set

[bhavesh0124]

I get this error

Error in 'delete' command: This command cannot be invoked after the command 'eventstats', which is not distributable streaming.

Anything else can i use here? my data is huge, i cant use a join or subsearch.

Thanks in advance

@ITWhisperer 

[PickleRick]

OK. What problem are you actually trying to solve here? And I'm not asking for the answer "I want to delete events". Why do you want to delete those events? What are those events? How did they get into your indexes? Why are you so eager to delete them (mind you they don't actually get deleted from disk, they just get marked as unreadable but are still present in the index files) instead of just letting them normally roll to frozen with time.

[ITWhisperer]

OK I don't use delete very often (nobody does), but you could try something like this

index=test [| search index=test | stats min(change_set) as change_set by source | format]

[bhavesh0124]

Thankyou @ITWhisperer .

Quick followup question, instead of index=test, i have a data model that has limited fields. Then how it would work?

| tstats dc(Changeset) as count, values(Changeset) as name_versions from datamodel=abc by source, name
| where count>1
| stats min(name_versions) by source, name

I want to delete now the events. How could i?

[ITWhisperer]

The delete command "deletes" the events which are in the pipeline - tstats does not return the events so this is unlikely to work. Having said that, I am not sure what happens if you do manage to delete the events from the index, whether the stats returned by tstats change?

[PickleRick]

It's a relatively old thread but I'll add my three cents.

Assuming we're pondering deleting events from the index based on which an accelerated datamodel summary is created (without DAS the answer to the question is obvious because the search from datamodel is simply translated to raw event search and executed against indexed events), as long as the deleted data is within the backfill range, I'd expect the summarization search to adjust the summary accordingly on the next scheduled run. If the deleted data falls into summary range but out of backfill range, I would expect the summary to stay untouched because there is no mechanism to update the summary.