Using Splunk 6.3.1. I have been given a list of about 2000 events that need to be "deleted" from my index. (I do understand that using |delete only hides the data...that is fine). I will be given a similar list at least weekly and possibly daily. Obviously I need to automate this, so I'm thinking a python script using the rest interface to execute a search. My question: what is the most efficient way to craft the search. If I were doing them one at a time, I would do
search index=foo identifier=theidentifierofmyevent | delete
But, I'm concerned that issuing 2,000 such commands would really tie up the search head. But, putting 2000 OR'd search terms possibly isn't the best idea either.
The file I receive is a text file with two space-separated columns, the first column is the field name and the second is the value to search for.
Any suggestions as to the best way to handle this? I guess as long as I wait for one search to finish, before starting another, it should be ok to do the searches one at a time. I just want to be as efficient as possible.
I would create a lookup table file from the text file OR ingest the text file, which contains identifiers, and then use a subsearch to extract those filters and delete. This you would be done in one search per day/week.
With lookup eventsToBeDeleted.csv contains one column identifier with 2000 rows
index=foo [| inputlookup eventsToBeDeleted.csv | table identifier] | delete
With text file ingested in SPlunk and field extractions are configure
index=foo [search index=something sourcetype=something | table identifier ] | delete