topic Re: events not deleting with | delete in Getting Data In

events not deleting with | delete

tawollen — Wed, 06 Oct 2010 21:47:41 GMT

I want to clear out information from a source that had bad information indexed, so I am trying to do a sourcetype=source | delete.

I do have can_delete role, the search deleted about 300,000 events, but there are 43 errors.

splunk_server index deleted errors
1 splunkserver ALL 304232 43
2 splunkserver main 304232

When I try the normal search, I still get the 43 events showing up. Re-running the delete command just comes back with the same 43 errors.

unfortunately, I can't clean the index that has this data. I still need some data from another source in that index.

If I try to drill down into the 43 errors, Splunk opens a new window with a flashtimeline and no search.

Re: events not deleting with | delete

tawollen — Wed, 06 Oct 2010 22:15:28 GMT

Interesting thing I found is that I had tried sending these events in as different sourcetypes, so the events (300,000) have been in Splunk as both "Type1" and "Type2".. When I do the | delete on Type2 I get the same errors. It turns out it looks like it the same 43 events will not delete out of either sourcetype.

This was a CSV file that was indexed, and the data source CSV file doesn't look strange for those lines that won't delete.

Re: events not deleting with | delete

Genti — Thu, 07 Oct 2010 06:28:56 GMT

Since you "desperately" want the data out of the index then you should be able to put some extra work and try this:
Export the data that you care for, csv format. Watch out for the size (rows) of the csv file.
Once you are sure you have all your data that you care for, roll the index and back up your data. Then, clean all the data for the index using the clean command for this index.

Then use inputcsv to get the data back into this clean index. Note again, only 10k data will be imported at once, so you will need to use a little trick to get the data in again.
see THIS for more info on importing large CSV files.

Re: events not deleting with | delete

Lowell — Thu, 07 Oct 2010 07:10:54 GMT

You can use the exporttool and importtool command line utilities to dump and recreate your index's bucket(s).

This approach lets you export an entire bucket to a single CSV file. You can then use a small little csv-processing script (or if your careful, a text editor) to strip out the offending lines before you re-create your bucket.

The process goes something like this:

cd $SPLUNK_HOME/var/lib/db/your_index
exportool db_xxx_xxx_id export_id.csv -csv
# Edit your export and remove unwanted events
vim export_id.csv
# ...
# Rebuild your bucket
importtool db_xxx_xxx_id.NEW export.csv
mv db_xxx_xxx_id db_xxx_xxx_id.OLD
mv db_xxx_xxx_id.NEW db_xxx_xxx_id
rm export_id.csv

WARNING: This is a potentially dangerous operation, you should backup and understand buckets and have a general idea of how indexing works before you try anything like this. You could shoot yourself in the foot. You have been warned.

BTW, if you are using event/block signing or anything like that, then you shouldn't try to attempt anything like this.

Also see:

http://www.splunk.com/wiki/Community:Modifying_indexed_data_via_export_and_import

Re: events not deleting with | delete

Lowell — Thu, 07 Oct 2010 07:16:57 GMT

What's importcsv? There's the search command inputcsv, and the command line tool importtool. Do you mean one of those?

Re: events not deleting with | delete

Genti — Fri, 08 Oct 2010 13:14:25 GMT

yeap, inputcsv editing the answer

Re: events not deleting with | delete

tawollen — Tue, 02 Nov 2010 22:43:09 GMT

We just left the data in the index and for future tests where we will be deleting data on a regular basis have created a 'test' index. With the test index we can run "./splunk clean eventdata test" that will clean that index completely so we can re-test an import. Having to do a csv export/import is more effort than we wanted to do. I am still not sure why splunk has issues causing errors when deleting (hiding) some data from an index with the | delete.