What are the performance impact or other ramifications of deleting events?

[the_wolverine]

We are trying out a method of deleting events from the index. It looks like a metadb rebuild takes place every time we do this so would like to know what the performance implications are, if any:

10-09-2012 16:01:13.287 +0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:01:38.986 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:05.529
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:02:30.585 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:56.595
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:03:23.114 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:03:48.600
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:04:14.583 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:04:40.996
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:05:06.330 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:05:32.331
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main'

[the_wolverine · Accepted solution]

From experience, it appears that DELETE is relatively expensive. When performing DELETE against many events (more than 10k), if done repeatedly in succession, for example, in the case of scheduled search that finds and "DELETE"s, bad things can happen if your system is undersized so watch it carefully. No idea what happens in a properly sized implementation but maybe Splunk can answer.