Splunk Search

What is a "scheduled_rtsearch?"



I'm configuring some new roles, and came across the "schedule_rtsearch" capability. The doc simply says "Lets the user schedule real-time saved searches." What is a scheduled rtsearch? Almost seems like an oxymoron.

Splunk Employee
Splunk Employee

In Splunk when you schedule a search you are provided with the option of scheduling a "Report" or an "Alert". The Alert gives you additional options to take some action (ie: send an email or run a script) when a trigger condition is met (ie: the search returns a count greater than 0). A scheduled rtsearch is really an alert which runs continuously realtime so a cron_schedule is irrelevant in this case.

When you create an alert in Splunk through the UI you set the alert type as either "Scheduled" or "Real-time". When you select Real-time, the scheduler will delegate the search and keep that search running continuously. This is a scheduled rtsearch.

You will see the sid of the search in resource_usage.log as : rt_scheduled....

dispatch.earliest_time defines how far back the rt searches looks over the data as it is running continuously so this is a sliding window. This can be configured in advanced settings of the alert or in the UI when you edit the alert under the trigger conditions > "in" x minutes/hours..etc field refers to the dispatch.earliest_time

"scheduling" it just means that if the node it is running on goes down/restarted or the search gets terminated, the scheduler will make sure it gets delegated to another member (if SHC) or respawned (if standalone SH) and makes sure the search is up and running again so the user does not need to intervene.

This is different than a real-time search in the traditional ad-hoc sense which get's killed when the user stops the search or closes the browser.

The Splunk Monitor Console provides a view of the search activity if you need to determine if a scheduled rtsearch is running:

from MC: >
Search>Activity>Search Activity: Deployment
Panel: Search Activity by Instance

Revered Legend

Esteemed Legend

Do not let ANYBODY have this capability, unless your product specifically (like ITSI) needs it. It is the best way to crush your Search Head.


Thanks. But, if you are scheduling a real-time search, how does that work? I would think that it would never end.

0 Karma

Splunk Employee
Splunk Employee

Essentially correct. I may be wrong but I always understood it to mean that it will start as per its scheduling and then run real-time until it's manually killed (I guess).

I think the bottom line is that you were correct in noticing it as an odd one and I usually don't allow any real time (rt) capabilities because they are rarely needed and can be so impactfull.

Get Updates on the Splunk Community!

Devesh Logendran, Splunk, and the Singapore Cyber Conquest

At this year’s Splunk University, I had the privilege of chatting with Devesh Logendran, one of the winners in ...

There's No Place Like Chrome and the Splunk Platform

WATCH NOW!Malware. Risky Extensions. Data Exfiltration. End-users are increasingly reliant on browsers to ...

Customer Experience | Join the Customer Advisory Board!

Are you ready to take your Splunk journey to the next level? 🚀 We invite you to join our elite squad ...