Splunk Enterprise Security

No results from scheduled jobs or searches

Dharmesh_93
Loves-to-Learn Lots

Hi,

We are using Splunk cloud 8.2 and mainly utilizing for Splunk SIEM solution. 

Currently we have many scheduled alerts, searches and reports. In the recent days we could see 21% of the searches were skipped and job execution time also increased. 

From yesterday, we are unable to see output results for any of the jobs, but we are getting the search result when we execute adhoc search.

We are also able to see below Errors and warnings in our console.

The percentage of non high priority searches skipped (74%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=7056. Total skipped Searches=5271

The instance is approaching the maximum number of historical searches that can be run concurrently.

The number of extremely lagged searches (1) over the last hour exceeded the red threshold (1) on this Splunk instance

Could you please share some solution to implement in this case. 

 

 

Labels (2)
0 Karma

PickleRick
Ultra Champion

Well, there's no real magic you can do. You have to lower the load or provide more resources.

In case of resources - In case of on-premise solution it would be just about adding indexers/search-heads to your environment and/or migrating to faster storage. I'm not sure how it works with cloud but I suspect it might have something to do with license "level".

You can (and should) also review the activity on your splunk deployment - see whether you really do need all those reports, searches, alerts and so on. And see if they are properly written and if they do not unnecessarily cover too big time ranges.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>