topic Re: |delete hang indefinetely in Getting Data In

|delete hang indefinetely

rbal_splunk — Mon, 28 Sep 2020 16:55:28 GMT

Issue :
I am trying to "delete" bunch of events from the index by using "| delete" command.
The search without "| delete" returns results with no problem. But when I add "| delete" to the end of the command line it just hangs.

Search query:
$ /opt/splunk/bin/splunk search 'index=security source=billing.log earliest=-39d latest=-38d'

Search query with "| delete"
/opt/splunk/bin/splunk search 'index=security source=billing.log earliest=-39d latest=-38d | delete'

The same query in the "Runnig jobs" list in UI looks like the following (http://goo.gl/8xITae):
index=security source=billing.log earliest=-39d latest=-38d | delete | head 100 | export add_timestamp=f add_offset=t segmentation=raw

Re: |delete hang indefinetely

rbal_splunk — Wed, 25 Jun 2014 06:30:19 GMT

It was found that bunch of .db_*.rbsentinel files have been created in the db/ folder. One of those files had a .lock extension, figured that the name of the *.lock file is related to the bucket that contains events which supposed to be deleted by the search query.

Had to stopped Splunk and removed all the .rbsentinel files and started splunk again. After that the same search query piped to the "delete" was able to delete events as expected.

Re: |delete hang indefinetely

sat94541 — Mon, 28 Sep 2020 16:55:30 GMT

This behavior is caused due to bugs

SPL-84276-If a search process is waiting to acquire a lock then it should print lock file information in search.log
SPL-81481-If a search process is waiting to acquire a lock then it should print lock file information in search.log

Bug is expected to be fixed din 6.0.6

Steps to reproduce

Search index=_internal sourcetype=splunk_web_access | eval bkt=_bkt | stats count by bkt, sourcetype and select a bucket drilldown on a slightly older bucket. For me, I had buckets 11 to 38 and I selected 35. Note the number of event that bucket
The drilldown search will look something like index=_internal sourcetype=splunk_web_access | eval bkt=_bkt | search bkt="_internal~35~687F80CD-2706-4248-8773-C09142624BB2". From the search results, select a unique requestid.
Run the search index=_internal sourcetype=splunk_web_access _bkt="35" 536a6a176e7fc09c5209d0 where 536a6a176e7fc09c5209d0 is my unique requestid and 35 is my selected bucket number. This should return only one result.
Now cd into the bucket var/lib/splunk/internaldb/db/db*35 and create a lock file with the same name as tsidx file. In my case the tsdix file name is 1399792823-1399758630-17977910403059112245.tsidx. The lock file created was 1399792823-1399758630-17977910403059112245.tsidx.lock
Now run index=_internal sourcetype=splunk_web_access _bkt="35" 536a6a176e7fc09c5209d0 | delete. This search should keep running indefinitely.
To replicate what the user behavior and get the matching pstacks, run the above search in another tab without closing this tab. Basically you have 2 searches running indefinetely

Re: |delete hang indefinetely

hogan24 — Tue, 08 Jul 2014 14:07:35 GMT

Are there any updates to this? I followed the steps above and deleted the mentioned files under "C:/Program Files/Splunk/var/lib/splunk/defaultdb/db" and restarted splunk. Then reran the |delete command and my delete is still hanging. Any additional help would be appreciated. Thanks.

Re: |delete hang indefinetely

hogan24 — Tue, 08 Jul 2014 14:10:02 GMT

I'm also currently on v6.1.1 if that helps. Thanks.

Re: |delete hang indefinetely

asetyyli — Fri, 10 Oct 2014 08:37:19 GMT

Removing the .rbsentinel files solved the issue for me. Thanks.