Getting Data In

Is it expected that the delete command will occasionally hang?

dmr195
Communicator

Last year somebody found a case of delete hanging indefinitely which could be cured by deleting .lock files from the index storage area. Somebody else answered that this was caused by bugs in Splunk: SPL-84276 and SPL-81481.

I too have seen the issue with delete hanging, and it is indeed curable by deleting .lock files as described in the other answer. I saw it with Splunk 6.2.1 and upgraded to 6.2.4 and haven't seen it since, but one of my colleagues just saw it today with Splunk 6.2.3.

I have a few questions (probably only answerable by Splunk insiders) about what the expected status of the delete command is. The title of bugs SPL-84276 and SPL-81481 was "If a search process is waiting to acquire a lock then it should print lock file information in search.log". To me that doesn't sound like the fix for these bugs actually solved the potential deadlock problem, just that affected searches should detect the deadlock and log it. Is that correct? If so, does anyone know the exact circumstances under which delete hangs? Is this considered a bug that will be fixed one day? Or is it already expected to be fixed, and if so, in which version of Splunk?

Tags (2)
0 Karma
1 Solution

lagnone_splunk
Splunk Employee
Splunk Employee

Hi dmr195,

SPL-84276 and the original SPL-81481 do not address the root cause of a | delete search changing, they only provide a means by which to identify why a search is in hung state.

The search.log of the search with the delete command will print that the search is blocked because of "xxxxxx-xxxxx.tsidx.lock" file. This can be used by support/the user to identify a stale lock file and delete it.

From what I can tell, the potential chance of this issue occurring is not strictly limited to searches with | delete.

View solution in original post

0 Karma

lagnone_splunk
Splunk Employee
Splunk Employee

Hi dmr195,

SPL-84276 and the original SPL-81481 do not address the root cause of a | delete search changing, they only provide a means by which to identify why a search is in hung state.

The search.log of the search with the delete command will print that the search is blocked because of "xxxxxx-xxxxx.tsidx.lock" file. This can be used by support/the user to identify a stale lock file and delete it.

From what I can tell, the potential chance of this issue occurring is not strictly limited to searches with | delete.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...