Deployment Architecture

How to t-shoot long-running queries that fail with " auto-canceled/failed due to an error/DAG Execution Exception"?

Glasses
Contributor

I have a user that kicks off long-running queries and complains that he gets job failures.

"DAG Execution Exception: Search has been cancelled"

"Search auto-canceled"

"The search job has failed due to an error. You may be able view job in the Job Inspector."

When I run the same query as admin, the job takes a while but it will complete without failure, usually around 20 minutes.

I check the shc ram usage and that looks fine,  the resources appear fine.

IDK if the user is running the search at high usage times, but I am not sure how to t-shoot where the issue is.

Any advice is appreciated.

Thank you

 

 

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

What version?

Does the user leave the search open or move to another tab or browser window?

This sounds like the job auto cancelling from the user losing the browser focus 

Glasses
Contributor

TY for the reply.

I will have to ask the user about behavior after launching the query.  But he did relay something weird, that being if he sets the job lifetime expiration to 7days then it does not time out.  I am not sure how job expiration would prevent the query timing out issue.  Unless he is mistaken.

If the user backgrounds the query would that prevent auto-cancel?

This user also runs these long queries via rest_api.

Is there  a way to set the auto-cancel time out to 60minutes for that role?

0 Karma

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

View solution in original post

0 Karma

Glasses
Contributor

Thank you, I will give it a try and update the post.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!