Monitoring Splunk

What are the performance impact or other ramifications of deleting events?

Champion

We are trying out a method of deleting events from the index. It looks like a metadb rebuild takes place every time we do this so would like to know what the performance implications are, if any:

10-09-2012 16:01:13.287 +0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:01:38.986 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:05.529
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:02:30.585 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:56.595
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:03:23.114 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:03:48.600
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:04:14.583 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:04:40.996
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:05:06.330 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:05:32.331
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main'
Tags (2)
0 Karma
1 Solution

Champion

From experience, it appears that DELETE is relatively expensive. When performing DELETE against many events (more than 10k), if done repeatedly in succession, for example, in the case of scheduled search that finds and "DELETE"s, bad things can happen if your system is undersized so watch it carefully. No idea what happens in a properly sized implementation but maybe Splunk can answer.

View solution in original post

0 Karma

Champion

From experience, it appears that DELETE is relatively expensive. When performing DELETE against many events (more than 10k), if done repeatedly in succession, for example, in the case of scheduled search that finds and "DELETE"s, bad things can happen if your system is undersized so watch it carefully. No idea what happens in a properly sized implementation but maybe Splunk can answer.

View solution in original post

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!