Getting error: "Dag Execution Exception: Search has been cancelled. Search auto-canceled".

[erez10121012]

Hi,

Every few minutes the dashboard shows me this error:

Dag Execution Exception: Search has been cancelled. Search auto-canceled.

I tried to change the dispatch.auto_cancel = 0, but the error returned.

Please help me find the solution for this error.

If it helps, this search is 30 seconds real time, and 1 min refresh.

Thanks.

[wyomoose]

Splunk now lists this as a known issue and offers a solution. Have not implemented it yet so can't vouch that it works.  We have the had the same issue for awhile. 

https://docs.splunk.com/Documentation/Splunk/8.2.4/ReleaseNotes/KnownIssues

[erez10121012]

i Understand that the problem is in the realtime search.

i change one of the dashboard gauge to 2 sec (not realtime) and the Execution  not shown on it.

but steel realtime is problem.

thanks

[salbro]

This also seems to be an issue with regular searches as well.

I have a search that looks at firewall data and gives me the top 10 source IP addresses. I converted it to a report on a dashboard that I leave up all day. All reports on the dashboard (9) are set to refresh every 5 minutes.

At least once a day, the same report gets the DAG execution failure. If I refresh the report, it doesnt load. I have to reload the page in order for it to resolve itself. None of the other reports on the dashboard do this.

[marceloalejandr]

salbro, 

did you ever get resolution to the "DAG Execution Exception..." error?

thanks in advance. 

[erez10121012]

No

just auto refresh the web page

[salbro]

No, I still get this daily on at least one panel of my dashboard. It occurs maybe 2-3 times a day, but usually the first time is after 4 hours.

The query is for 15 minutes worth of firewall data, looks at IP location, filters against a manual blacklist of IP addresses kept in a CSV, and shows the top 10 external hosts. There is a refresh of 15 minutes on the search.

If I see the DAG failure message. I stop the panel and refresh the dashboard.

[twollenslegel_s]

This is sort of a known issue. What likely happened is you started a search, then changed tabs within a browser. . If you are getting this message, it is best to stay in the same tab and let the page load fully. This should help you a bit. 

[salbro]

This will help some, but ultimately I believe it will remain an issue. I have 3 monitors and use one as a dedicated Splunk dashboard screen. The panels auto-refresh every 5-15 minutes, and I don't maintain focus on the dashboard so I can work on other things.

I will say that after converting this over into Dashboard Studio (absolute mode), this hasn't happened once.