we got some Events, which we need to clean up. So we need to wipe them:
$HOME/bin/splunk search 'index=index kpi_type=voldemort earliest=09/01/2016:00:00:00 | delete ' -auth username:XXXXXXXX
But instead auf marking them to deleted. I get:
ERROR: 7074012 event could not be deleted INFO: 0 events successfully deleted INFO: Your timerange was substituted based on your search string splunk_server index deleted errors ------------- ------- ------- ------- b23 __ALL__ 0 440674 b25 __ALL__ 0 2253332 b26 __ALL__ 0 1461429 idx-05 __ALL__ 0 1047879 idx-06 __ALL__ 0 451062 s574 __ALL__ 0 1419636
A Event looks like this:
timestamp, offers_position=1.000000, number_of_offers=1.000000, product_id=999967, offers_shop_id=285850, index=voldemort, leadouts=1, category_id=10032, leadouts_gesamt=1, kpi_type=voldemort
I dont see any Errors in either Indexer-Splunkd.log or Searchhead splunkd.log
Its not a permission issue(my role has the can_delete role imported). Also the search.log shows Only something like "cant delete" no explicit error.
I also tried using another Searchhead and the Web-Interface.
Has anyone a clue?
The upgrade to Splunk> 6.4.3 from 6.1.1 brought no change 😞
Have you added username to the candelete role or granted the deleteby_keyword capability?
By default nobody (including admin) has that:
Have you also tried running your query from the UI instead of the CLI?
Are you sure that your user role has the correct permission to delete events? usually Admin doesn't have this permissions, only "can_delete" user has this permission!
Remeber that the delete command makes a logical and not a physical deletion, so you don't free any disk space (see https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Delete).
To physically delete events you can only clean an intere index (see http://docs.splunk.com/Documentation/Splunk/6.4.3/Indexer/RemovedatafromSplunk).
You can verify accessing role capabilities [Settings -- Access Controls -- Roles -- Admin].
Try using web interface and user can_delete.
Got an update on this.
The problem is the field "index" in the Eventdata. This causes an issue for splunk.
To resolv this issue you have to evaluate the splunk-index-field.
index=nameofindex kpi_type=voldemort earliest=09/01/2016:00:00:00| eval index= "nameofindex" | delete
I could delete everything successful.
Good point, and it is documented in delete command documentation (https://docs.splunk.com/Documentation/Splunk/6.6.0/SearchReference/Delete)
Note: The delete command does not work if your events contain a field named index aside from the default index field that is applied to all events. If your events do contain an additional index field, you can use eval before invoking delete, as in this example: index=fbus_summary latest=1417356000 earliest=1417273200 | eval index = "fbus_summary" | delete