SHC Concurrency Search control

[0xAli]

Hi,
I hope all is well.

I am writting to ensure i am getting the correct picture on the SHC search:

Assumption: 
3x search head with CPU: 32 cores.
max search per cpu = 2

1. max_hist_searches =  max_searches_per_cpu x number_of_cpus + base_max_searches (our environment is 70 per node)
2. Max Scheduled Search = 50% * max_hist_searches  = 35 
3. Max Summarization  = 50% * Scheduled Search =17
4. Max Ad hoc guaranteed = 35 Remaining — always reserved for analysts
   
   Only SCHEDULED searches are coordinated by the captain across nodes
   
   
   Reference:
   https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/10.4/manage-search-head-…
   https://community.splunk.com/t5/Reporting/How-to-calculate-Splunk-search-concurrency-limit-for-hist…
   Are You Skipping? Please Read! | Splunk
   
   

   For Real-time searches:
   max_rt_searches = max_rt_search_multiplier x max_hist_searches

   1. Max real-time searches = 1 * 70 = 70 ( our environment per node).
   2. Max real-time scheduled searches = 50% x 70 = 35
   3. Max Ad hoc guaranteed = 35 Remaining — always reserved for analysts

 As per the above, we can conclude the below per each node:

• Historical ad hoc → shared 70 pool (User manuall search past data)
• Historical scheduled → 35 (50% of 70) 
• Summarization → 17 (50% of 35) (Datamodel)
• RT scheduled → 35 (separate RT pool, independent) 
• RT ad hoc → 35 (RT guaranteed, reserved for analysts) (User manually search for RT live stream data)
• Scheduled ad hoc → does not exist

[0xAli]

Guys, 

Please read these articles carefully:

1. https://help.splunk.com/en/splunk-enterprise/search/search-manual/9.3/search-and-report-in-real-tim…
2. https://help.splunk.com/en/splunk-enterprise/administer/distributed-search/9.4/manage-search-head-c…
3. https://splunk.my.site.com/customer/s/article/Real-time-scheduling-vs-Continuous-scheduling
   
   Summary:
   
   the type of the scheduled search:
   The Earliest and Latest time attributes are determine the type of the scheduled search weather it's Real-Time OR Historical
   
   Rule:
          Search for the past data = Historical  -5m@m to -65m@m
          Search for the Live stream = Real-Time rt-5m to rt-65m
   
   the scheduling mode: 
   determine what is the behavior of the scheduled search when no available search slots, if it will be skipped or deferred.
   
   Rule:
           Continuous mode: Deferred
            Real-Time mode: Skipped
   
    

[0xAli]

Kindly, from the below screenshots, we can find that there are some RT scheduled search are enabled by default by splunk itself, and use the indexed real-time, so can we create some real-time scheduled search so we can accomodate the RT pool?

[PickleRick]

No.  If you turn on indexed real-time searches, it changes slightly the way real-time searches work. But it's still about real-time search, not about real-time schedule. (yes, I know it's confusing, I've already said so ;-))

[0xAli]

Thank you, really appreciated!

[gcusello]

Hi @0xAli ,

good for you, see next time!

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello]

Hi @0xAli ,

your settings seem to be correct, the only way to proceed is to analyze the load on your SHs using the Monitoring Consle, in this way you can see the health status of your system and eventually put some tuning action, e.g.:

• check the scheduling of the scheduled searches,
• avoid to use real-time searches and transforms them in scheduled searches,
• optimize your custom searches,
• check the performances of the Data Models,
• etc...

Ciao.

Giuseppe

[0xAli]

Thanks for your support,  it's really appreciated!

Could you please clarify more on that point:

"avoid to use real-time searches and transforms them in scheduled searches,"

As from the calculated values, we have two separate pool, one for the historical scheduled search(35 - Summarization), and one for the RT-scheduled search(35), so the capacity of the RT will not affect the historical, and it's higher than historical (35 = Summarization + Historical scheduled search).

real-time searches meand ES detection + RT mode ?
 scheduled searches means ES detection + Continous mode ?

[gcusello]

Hi @0xAli ,

about first question:

in Splunk a search takes a CPU (more if you have also subsearches in the main search) until it will finish, so a RT search takes one or more CPUs for always, it's better to schedule a search so it will release the CPUs when finised.

About second question:

No RT depends on how you configured your detections: scheduled or RT, as I said if you can avoid RT, otherwise give more resources (CPUs to your Indexers and Search Heads)!

On Splunk Cloud RT searches are usually blocked for all not admin users!

Ciao.

Giuseppe

[0xAli]

Now, i got your point, the confuse comes from the hint under the RT/scheduled mode

[PickleRick · Accepted solution]

Yes, the terminology here can be somewhat confusing. Splunk uses the term "real-time" in two different contexts when it comes to searching. 

One thing is a real time search. It is a search which is done not against already indexed data but on the incoming data as it is ingested by Splunk. This type of search allocates one cpu on a search head running the search for the whole time the search is running and one cpu on each participating indexer. The other type of a search is a historical search which runs on data returned by indexers from the buckets residing on disk - already indexed data.

Another thing is the realtime schedule mode. This means that a run of a scheduled search will be attempted at the scheduled time but if there are no free search slots, Splunk will delay it for some time (if configuration of the search allows it) but if it still cannot find free slot to run it the search run will get skipped. The other scheduling mode is continuous which means that Splunk will try to run the search for a given time slot indefinitely until it finally can do so (I suppose there are some technical limits to that but that's the general idea). The caveat is that continuously scheduled searches have lower priority within the scheduler.

[0xAli]

Thank you!

Let me leave what i have got from the truly insighful discussion with you all below:

1. Schedule Mode (Continous or real-time) doesn't determine the type of the Scheduled Search(RTScheduled Search OR Historical Scheduled Search ), it just determine the behaviour at the status: unavailable search slots, and how the Scheduled Search will run weather it will be skipped or DEFERRED?

then historical scheduled Search can be operate at the Continuous mode ot the Real-time mode ?
A Real-time scheduled Search always uses Real-time scheduling by nature?

If am i correct - as per the above - kindly, give me example on the Real-time scheduled Search and historical scheduled Search (when can i say it's RT-scheduled Search or historical scheduled Search)?

All event-based detection is historical scheduled search?

Thank in advance!

[gcusello]

Hi @0xAli ,

about the first assumption, it isn't correct:

Schedule Mode (Continous or real-time) determine the type of the Scheduled Search(RTScheduled Search OR Historical Scheduled Search)

historical scheduled Search can be operate at the Continuous mode ot the Real-time mode ?

historical scheduled Search are in Continuous Mode, not in RT Mode, what's the sense of a RT historical search?

A Real-time scheduled Search always uses Real-time scheduling by nature?

yes

Ciao.

Giuseppe

[gcusello]

Hi @0xAli ,

is is clear for you or do you need other information?

Ciao.

Giuseppe