Splunk Enterprise Security

No results from scheduled jobs or searches

Dharmesh_93
Loves-to-Learn Lots

Hi,

We are using Splunk cloud 8.2 and mainly utilizing for Splunk SIEM solution. 

Currently we have many scheduled alerts, searches and reports. In the recent days we could see 21% of the searches were skipped and job execution time also increased. 

From yesterday, we are unable to see output results for any of the jobs, but we are getting the search result when we execute adhoc search.

We are also able to see below Errors and warnings in our console.

The percentage of non high priority searches skipped (74%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=7056. Total skipped Searches=5271

The instance is approaching the maximum number of historical searches that can be run concurrently.

The number of extremely lagged searches (1) over the last hour exceeded the red threshold (1) on this Splunk instance

Could you please share some solution to implement in this case. 

 

 

Labels (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Well, there's no real magic you can do. You have to lower the load or provide more resources.

In case of resources - In case of on-premise solution it would be just about adding indexers/search-heads to your environment and/or migrating to faster storage. I'm not sure how it works with cloud but I suspect it might have something to do with license "level".

You can (and should) also review the activity on your splunk deployment - see whether you really do need all those reports, searches, alerts and so on. And see if they are properly written and if they do not unnecessarily cover too big time ranges.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...

Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend ...

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward ...