We have a platform where lot of dashboards are populated using splunk searches via splunk api call.
All the query runs fine and brings results in 30-40 secs however there is one query which brings results in around 5 mins. The query is not complex and runs within seconds on splunk search heads and I also used postman to check how long it takes- it was around seconds on postman as well. I beleive there is no issue at query end.
I checked the splunk logs for that particular SID which took 5 mins to run: I see below ERROR logs: