Need help crafting an efficent solution for deleting 2000 events...

[lyndac]

Using Splunk 6.3.1. I have been given a list of about 2000 events that need to be "deleted" from my index. (I do understand that using |delete only hides the data...that is fine). I will be given a similar list at least weekly and possibly daily. Obviously I need to automate this, so I'm thinking a python script using the rest interface to execute a search. My question: what is the most efficient way to craft the search. If I were doing them one at a time, I would do

search index=foo identifier=theidentifierofmyevent | delete

But, I'm concerned that issuing 2,000 such commands would really tie up the search head. But, putting 2000 OR'd search terms possibly isn't the best idea either.

The file I receive is a text file with two space-separated columns, the first column is the field name and the second is the value to search for.

Any suggestions as to the best way to handle this? I guess as long as I wait for one search to finish, before starting another, it should be ok to do the searches one at a time. I just want to be as efficient as possible.

[somesoni2 · Accepted solution]

I would create a lookup table file from the text file OR ingest the text file, which contains identifiers, and then use a subsearch to extract those filters and delete. This you would be done in one search per day/week.

sample search
With lookup eventsToBeDeleted.csv contains one column identifier with 2000 rows

index=foo [| inputlookup eventsToBeDeleted.csv | table identifier] | delete

With text file ingested in SPlunk and field extractions are configure

index=foo [search index=something sourcetype=something | table identifier ] | delete

[jkat54]

Honestly I would go with the OR approach so that it's one search.