Deployment Architecture

How to t-shoot long-running queries that fail with " auto-canceled/failed due to an error/DAG Execution Exception"?

Glasses
Builder

I have a user that kicks off long-running queries and complains that he gets job failures.

"DAG Execution Exception: Search has been cancelled"

"Search auto-canceled"

"The search job has failed due to an error. You may be able view job in the Job Inspector."

When I run the same query as admin, the job takes a while but it will complete without failure, usually around 20 minutes.

I check the shc ram usage and that looks fine,  the resources appear fine.

IDK if the user is running the search at high usage times, but I am not sure how to t-shoot where the issue is.

Any advice is appreciated.

Thank you

 

 

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

View solution in original post

gjanders
SplunkTrust
SplunkTrust

What version?

Does the user leave the search open or move to another tab or browser window?

This sounds like the job auto cancelling from the user losing the browser focus 

Glasses
Builder

TY for the reply.

I will have to ask the user about behavior after launching the query.  But he did relay something weird, that being if he sets the job lifetime expiration to 7days then it does not time out.  I am not sure how job expiration would prevent the query timing out issue.  Unless he is mistaken.

If the user backgrounds the query would that prevent auto-cancel?

This user also runs these long queries via rest_api.

Is there  a way to set the auto-cancel time out to 60minutes for that role?

0 Karma

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

Glasses
Builder

Thank you, I will give it a try and update the post.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...