Can't delete Events

[effem]

Hello,

we got some Events, which we need to clean up. So we need to wipe them:

$HOME/bin/splunk search 'index=index kpi_type=voldemort earliest=09/01/2016:00:00:00 | delete ' -auth username:XXXXXXXX

But instead auf marking them to deleted. I get:

ERROR: 7074012 event could not be deleted
INFO: 0 events successfully deleted
INFO: Your timerange was substituted based on your search string
splunk_server  index  deleted errors
------------- ------- ------- -------
b23           __ALL__       0  440674
b25           __ALL__       0 2253332
b26           __ALL__       0 1461429
idx-05        __ALL__       0 1047879
idx-06        __ALL__       0  451062
s574          __ALL__       0 1419636

A Event looks like this:

timestamp, offers_position=1.000000, number_of_offers=1.000000, product_id=999967, offers_shop_id=285850, index=voldemort, leadouts=1, category_id=10032, leadouts_gesamt=1, kpi_type=voldemort

I dont see any Errors in either Indexer-Splunkd.log or Searchhead splunkd.log
Its not a permission issue(my role has the can_delete role imported). Also the search.log shows Only something like "cant delete" no explicit error.

I also tried using another Searchhead and the Web-Interface.

Has anyone a clue?

Update
The upgrade to Splunk> 6.4.3 from 6.1.1 brought no change 😞

[effem · Accepted solution]

Hello.
Got an update on this.

The problem is the field "index" in the Eventdata. This causes an issue for splunk.
To resolv this issue you have to evaluate the splunk-index-field.

index=nameofindex kpi_type=voldemort earliest=09/01/2016:00:00:00| eval index= "nameofindex" | delete

I could delete everything successful.

[aakwah]

Good point, and it is documented in delete command documentation (https://docs.splunk.com/Documentation/Splunk/6.6.0/SearchReference/Delete)

Note: The delete command does not work if your events contain a field named index aside from the default index field that is applied to all events. If your events do contain an additional index field, you can use eval before invoking delete, as in this example:
index=fbus_summary latest=1417356000 earliest=1417273200 | eval index = "fbus_summary" | delete

Regards

[gcusello]

Are you sure that your user role has the correct permission to delete events? usually Admin doesn't have this permissions, only "can_delete" user has this permission!
Remeber that the delete command makes a logical and not a physical deletion, so you don't free any disk space (see https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Delete).
To physically delete events you can only clean an intere index (see http://docs.splunk.com/Documentation/Splunk/6.4.3/Indexer/RemovedatafromSplunk).
Bye.
Giuseppe

[effem]

cleaning the index is not an option. And im very sure it is not a permission issue.

[gcusello]

You can verify accessing role capabilities [Settings -- Access Controls -- Roles -- Admin].
Try using web interface and user can_delete.
Bye.
Giuseppe

[effem]

I already made sure i got the permissions. As i said. It is not a permissions issue.

[javiergn]

Have you added username to the can_delete role or granted the delete_by_keyword capability?
By default nobody (including admin) has that:

https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Delete#Usage

Have you also tried running your query from the UI instead of the CLI?

[effem]

I tried it via UI also. And as stated it is not a permission issue.