Alerting

Splunk "real-time" using system resources ?

SS1
Path Finder

Hello SPlunkers,

'real-time' alerts are using up the maximum resources and resulting in skipped searches. does using a cron schedule which runs every 1 minute and the time range of last 61 seconds also cause stress to the system. What would be the ideal time range to use ?

 

SS1_0-1612220063542.png

 

Labels (4)
0 Karma

scelikok
SplunkTrust
SplunkTrust

Every search effects system performance but while it is running. But changing from realtime to scheduled will help for sure.

Each realtime search keeps one CPU core all the time, that is why may causes skipped searches if search concurrency is high. When you change search to scheduled, it will release resources after search finalize. Splunk will be able to use those resource for other searches or processes. 

If you do not need to get that alert immediately, higher is better. It is better to think your response time to alert, if you cannot act immediately there maybe no need to run search realtime or every minute.

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @SS1,

If your search does not take more than a minute, scheduled mode will help you a lot. You can use below crontab to run every minute;

* * * * *

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

SS1
Path Finder

My question is will it affect the perfomance of the splunk system if the cron runs every min?

 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

ATTENTION: We’re Moving! (AGAIN!)

The Splunk Community Slack is undergoing a system migration to keep our workspace secure and ...

Deep Dive: Optimizing Telemetry Pipelines in Splunk Observability Cloud

In this session, we will peel back the layers of Splunk Observability Cloud’s cost-optimization features. ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...