Splunk Search

Why do real time searches create many rt_scheduler_* directories, how can I control them

chris
Motivator

I have set up a single real time alert that creates about 1000 rt_scheduler__ entries in /var/run/splunk/dispatch/. Is there a possibility control the amount of directories that are created (is this related to the ttl of the search)?
Otherwise I will have to increase the dispatch_dir_warning_size in limits.conf which is not really a solution if I configure additional alerts.

[rtalert_nevis_err_requests_1min]
action.email = 1
action.email.inline = 1
action.email.reportServerEnabled = 0
action.email.sendresults = 1
action.email.to = me@me.com
alert.digest_mode = False
alert.expires = 6h
alert.suppress = 1
alert.suppress.fields = host
alert.suppress.period = 30m
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = rt-1m
dispatch.latest_time = rt
displayview = flashtimeline
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_view = flashtimeline
search = sourcetype="proxy"   | stats sum(req) as req sum(req_4xx) as req_4xx sum(req_5xx) as req_5xx by host | eval error_rate=if(req==0,0,round((req_4xx+req_5xx)/req,3)) | where error_rate>0,5 
Tags (2)
1 Solution

chris
Motivator

I had to set the alert condition of the alerts to something different that "always". This prevents splunk from creating a directory every time the alert is run/triggered which is a lot for rt alerts.

View solution in original post

0 Karma

chris
Motivator

I had to set the alert condition of the alerts to something different that "always". This prevents splunk from creating a directory every time the alert is run/triggered which is a lot for rt alerts.

0 Karma

sourabh_varshne
Explorer

Hi Chris,You dnt have option other than to manually delete the data from your dispatch directory if your alert creates 1000 entries . This is taken care by default from Splunk only.

0 Karma

chris
Motivator

Thank you for taking time to reply, manually deleting the directories does not solve the problem

0 Karma

sourabh_varshne
Explorer

Hi Chris,

You dnt have option other than to manually delete the data from your dispatch directory if your alert creates 1000 entries . This is taken care by default from Splunk only.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...