Splunk Search

What is a "scheduled_rtsearch?"

a212830
Champion

Hi,

I'm configuring some new roles, and came across the "schedule_rtsearch" capability. The doc simply says "Lets the user schedule real-time saved searches." What is a scheduled rtsearch? Almost seems like an oxymoron.

rphillips_splk
Splunk Employee
Splunk Employee

In Splunk when you schedule a search you are provided with the option of scheduling a "Report" or an "Alert". The Alert gives you additional options to take some action (ie: send an email or run a script) when a trigger condition is met (ie: the search returns a count greater than 0). A scheduled rtsearch is really an alert which runs continuously realtime so a cron_schedule is irrelevant in this case.

When you create an alert in Splunk through the UI you set the alert type as either "Scheduled" or "Real-time". When you select Real-time, the scheduler will delegate the search and keep that search running continuously. This is a scheduled rtsearch.

You will see the sid of the search in resource_usage.log as : rt_scheduled....

dispatch.earliest_time defines how far back the rt searches looks over the data as it is running continuously so this is a sliding window. This can be configured in advanced settings of the alert or in the UI when you edit the alert under the trigger conditions > "in" x minutes/hours..etc field refers to the dispatch.earliest_time

"scheduling" it just means that if the node it is running on goes down/restarted or the search gets terminated, the scheduler will make sure it gets delegated to another member (if SHC) or respawned (if standalone SH) and makes sure the search is up and running again so the user does not need to intervene.

This is different than a real-time search in the traditional ad-hoc sense which get's killed when the user stops the search or closes the browser.

The Splunk Monitor Console provides a view of the search activity if you need to determine if a scheduled rtsearch is running:

from MC: >
Search>Activity>Search Activity: Deployment
Panel: Search Activity by Instance

somesoni2
Revered Legend

woodcock
Esteemed Legend

Do not let ANYBODY have this capability, unless your product specifically (like ITSI) needs it. It is the best way to crush your Search Head.

Sid
Explorer

apologies @woodcock i am too late here but would it work with a role not having rtsearch capability  ?
doc says 

schedule_rtsearchLets the user schedule real-time saved searches. The schedule_search and rtsearch capabilities must also be assigned to the role.
0 Karma

a212830
Champion

Thanks. But, if you are scheduling a real-time search, how does that work? I would think that it would never end.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Essentially correct. I may be wrong but I always understood it to mean that it will start as per its scheduling and then run real-time until it's manually killed (I guess).

I think the bottom line is that you were correct in noticing it as an odd one and I usually don't allow any real time (rt) capabilities because they are rarely needed and can be so impactfull.

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...