No results from scheduled jobs or searches

[Dharmesh_93]

Hi,

We are using Splunk cloud 8.2 and mainly utilizing for Splunk SIEM solution. 

Currently we have many scheduled alerts, searches and reports. In the recent days we could see 21% of the searches were skipped and job execution time also increased. 

From yesterday, we are unable to see output results for any of the jobs, but we are getting the search result when we execute adhoc search.

We are also able to see below Errors and warnings in our console.

The percentage of non high priority searches skipped (74%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=7056. Total skipped Searches=5271

The instance is approaching the maximum number of historical searches that can be run concurrently.

The number of extremely lagged searches (1) over the last hour exceeded the red threshold (1) on this Splunk instance

Could you please share some solution to implement in this case. 

[PickleRick]

Well, there's no real magic you can do. You have to lower the load or provide more resources.

In case of resources - In case of on-premise solution it would be just about adding indexers/search-heads to your environment and/or migrating to faster storage. I'm not sure how it works with cloud but I suspect it might have something to do with license "level".

You can (and should) also review the activity on your splunk deployment - see whether you really do need all those reports, searches, alerts and so on. And see if they are properly written and if they do not unnecessarily cover too big time ranges.