Deployment Architecture

How to t-shoot long-running queries that fail with " auto-canceled/failed due to an error/DAG Execution Exception"?

Glasses
Builder

I have a user that kicks off long-running queries and complains that he gets job failures.

"DAG Execution Exception: Search has been cancelled"

"Search auto-canceled"

"The search job has failed due to an error. You may be able view job in the Job Inspector."

When I run the same query as admin, the job takes a while but it will complete without failure, usually around 20 minutes.

I check the shc ram usage and that looks fine,  the resources appear fine.

IDK if the user is running the search at high usage times, but I am not sure how to t-shoot where the issue is.

Any advice is appreciated.

Thank you

 

 

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

View solution in original post

gjanders
SplunkTrust
SplunkTrust

What version?

Does the user leave the search open or move to another tab or browser window?

This sounds like the job auto cancelling from the user losing the browser focus 

Glasses
Builder

TY for the reply.

I will have to ask the user about behavior after launching the query.  But he did relay something weird, that being if he sets the job lifetime expiration to 7days then it does not time out.  I am not sure how job expiration would prevent the query timing out issue.  Unless he is mistaken.

If the user backgrounds the query would that prevent auto-cancel?

This user also runs these long queries via rest_api.

Is there  a way to set the auto-cancel time out to 60minutes for that role?

0 Karma

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

Glasses
Builder

Thank you, I will give it a try and update the post.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...