Splunk Search

Why do real time searches create many rt_scheduler_* directories, how can I control them

chris
Motivator

I have set up a single real time alert that creates about 1000 rt_scheduler__ entries in /var/run/splunk/dispatch/. Is there a possibility control the amount of directories that are created (is this related to the ttl of the search)?
Otherwise I will have to increase the dispatch_dir_warning_size in limits.conf which is not really a solution if I configure additional alerts.

[rtalert_nevis_err_requests_1min]
action.email = 1
action.email.inline = 1
action.email.reportServerEnabled = 0
action.email.sendresults = 1
action.email.to = me@me.com
alert.digest_mode = False
alert.expires = 6h
alert.suppress = 1
alert.suppress.fields = host
alert.suppress.period = 30m
alert.track = 0
cron_schedule = * * * * *
dispatch.earliest_time = rt-1m
dispatch.latest_time = rt
displayview = flashtimeline
enableSched = 1
quantity = 0
relation = greater than
request.ui_dispatch_view = flashtimeline
search = sourcetype="proxy"   | stats sum(req) as req sum(req_4xx) as req_4xx sum(req_5xx) as req_5xx by host | eval error_rate=if(req==0,0,round((req_4xx+req_5xx)/req,3)) | where error_rate>0,5 
Tags (2)
1 Solution

chris
Motivator

I had to set the alert condition of the alerts to something different that "always". This prevents splunk from creating a directory every time the alert is run/triggered which is a lot for rt alerts.

View solution in original post

0 Karma

chris
Motivator

I had to set the alert condition of the alerts to something different that "always". This prevents splunk from creating a directory every time the alert is run/triggered which is a lot for rt alerts.

0 Karma

sourabh_varshne
Explorer

Hi Chris,You dnt have option other than to manually delete the data from your dispatch directory if your alert creates 1000 entries . This is taken care by default from Splunk only.

0 Karma

chris
Motivator

Thank you for taking time to reply, manually deleting the directories does not solve the problem

0 Karma

sourabh_varshne
Explorer

Hi Chris,

You dnt have option other than to manually delete the data from your dispatch directory if your alert creates 1000 entries . This is taken care by default from Splunk only.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...