Deployment Architecture

How to t-shoot long-running queries that fail with " auto-canceled/failed due to an error/DAG Execution Exception"?

Glasses
Builder

I have a user that kicks off long-running queries and complains that he gets job failures.

"DAG Execution Exception: Search has been cancelled"

"Search auto-canceled"

"The search job has failed due to an error. You may be able view job in the Job Inspector."

When I run the same query as admin, the job takes a while but it will complete without failure, usually around 20 minutes.

I check the shc ram usage and that looks fine,  the resources appear fine.

IDK if the user is running the search at high usage times, but I am not sure how to t-shoot where the issue is.

Any advice is appreciated.

Thank you

 

 

Labels (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

View solution in original post

gjanders
SplunkTrust
SplunkTrust

What version?

Does the user leave the search open or move to another tab or browser window?

This sounds like the job auto cancelling from the user losing the browser focus 

Glasses
Builder

TY for the reply.

I will have to ask the user about behavior after launching the query.  But he did relay something weird, that being if he sets the job lifetime expiration to 7days then it does not time out.  I am not sure how job expiration would prevent the query timing out issue.  Unless he is mistaken.

If the user backgrounds the query would that prevent auto-cancel?

This user also runs these long queries via rest_api.

Is there  a way to set the auto-cancel time out to 60minutes for that role?

0 Karma

gjanders
SplunkTrust
SplunkTrust

Send to background always fixes it for me. I'm unsure where the timeout is controlled 

Glasses
Builder

Thank you, I will give it a try and update the post.

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...