How does Splunk components and infrastructure work under scheduled search?

[Taruchit]

Hello All,

I have a SPL which is scheduled to run each minute for a span of 1 hour.

On each execution the search runs for 4 seconds with size of around 400KB.

Thus, how does the scheduler and search head work in such scenario at the backend? Does the scheduled SPL keeps scheduler and search head busy for entire 1 hour? Or they are free to run the other SPLs during that span of 1 hour?

And can you share any negative implications on Splunk infrastructure due to the above scheduled search?

Any information would be very helpful.

Thank you

Taruchit

[inventsekar]

Hi @Taruchit ... Some details about Splunk searches:

On the Monitoring Console, you can run a healthcheck, which will tell you how the systems performance looks like. 

The limits.conf file got settings for controlling how many searches the search head can run(for all types - real time, historical, concurrent, etc)

https://docs.splunk.com/Documentation/Splunk/9.1.1/admin/limitsconf#limits.conf.example

[scheduler]

# Percent of total concurrent searches that will be used by scheduler is
# total concurrency x max_searches_perc = 20 x 60% = 12 scheduled searches
# User default value (needed only if different from system/default value) when
# no max_searches_perc.<n>.when (if any) below matches.
max_searches_perc = 60

# Increase the value between midnight-5AM.
max_searches_perc.0 = 75
max_searches_perc.0.when = * 0-5 * * *

# More specifically, increase it even more on weekends.
max_searches_perc.1 = 85
max_searches_perc.1.when = * 0-5 * * 0,6

# Maximum number of concurrent searches is enforced cluster-wide by the
# captain for scheduled searches. For a 3 node SHC total concurrent
# searches = 3 x 20 = 60. The total searches (adhoc + scheduled) = 60, then
# no more scheduled searches can start until some slots are free.
shc_syswide_quota_enforcement = true

[Taruchit]

Hello @inventsekar,

Thank you for your response and for sharing the details.

I searched about healthcheckup and found it gives information related to configuration changes.

I check Job Details dashboard and fetched details of average indexer time which is around 0.9 seconds and time taken by the search to run on each indexer which varied between 0.8 to 1.5 seconds. Thus, can you please share if its suitable to run the scheduled search over Splunk infrastructure or if it can lead to negative implications and issues? The reason to ask is because its scheduled search and cannot afford search failure or infrastructure breakdown, and thus, want to be sure of using the approach.

Thank you

[inventsekar]

>>> I have a SPL which is scheduled to run each minute for a span of 1 hour.

>>>On each execution the search runs for 4 seconds with size of around 400KB.

>>>Thus, how does the scheduler and search head work in such scenario at the backend? Does the scheduled SPL keeps scheduler and search head busy for entire 1 hour? Or they are free to run the other SPLs during that span of 1 hour?

this looks like a normal search. Please update us... is it a clustered environment?

if possible, share us the Splunk search query(pls remove hostnames, ip address, any sensitive info from the search query before posting it here), so that we can fine-tune it. so you can make sure of there will be no negative impacts. 

one better idea - did you consider the "Summary indexing"

https://docs.splunk.com/Documentation/Splunk/9.1.1/Knowledge/Configuresummaryindexes

one more doc for your reference:

https://www.splunk.com/en_us/blog/customers/splunk-clara-fication-search-best-practices.html?locale=...

>>>And can you share any negative implications on Splunk infrastructure due to the above scheduled search?

this looks like a doable job and there should be no negative implications.
Splunk should be able to handle this scheduled search easily. 

the best approach is to have a UAT / Dev / Test Splunk environment(as much as possible, it should replicate the production Splunk environment) and run this search on the Uat/Dev/Test Splunk first. so that you can get assurance and clear confirmation before running something on the production Splunk. 

[Taruchit]

Hello @inventsekar,

Thank you for your response and for sharing the details.

1. Yes, I used summary indexing to improve the performance of the search query.

2. Yes, the Splunk infrastructure is a clustered environment.

Thank you for sharing the documents for improving search performance and for sharing that Splunk can run the search without any negative implications or major issues.

Taruchit

[isoutamo · Accepted solution]

Hi

As you have scheduled as a historic (not real time) search it's quite ok. It reserved those resources only that 4s time per each minute for that SPL. If it was a real time (basically you never need that) then it reserve 1cpu for all time from all nodes (SH + IDXs) what you have on your environment which are participating that query.

Then it's totally another question is that every minute schedule something what you are needing? You should consider how important it's to get that alert and how fast you can react and fix the issue.

Splunk is not an infrastructure monitor system, even you can use it for that! There are many other tools including Spunk IM which are better for that purpose.

r. Ismo

[Taruchit]

Hello @isoutamo,

Thank you for your response.

The scheduled SPL is a report scheduled to run each minute to fetch datapoints for different conditions in each iteration and store them in lookup file so that later I can use the lookup file to directly read data for my analysis and also in dashboards and alerts. This helps to access data faster and load dashboard with all relevant information with lesser time than reading and displaying the data from index. This also helps to access required historical data faster than fetching from index.

So as I understand from your inputs, in my current situation all relevant Splunk components are reserved for the scheduled search each minute for 4 seconds for the period of 1 hour. And thus, it works as a normal historical scheduled search with no extra resource workload or causing slowness, breakdowns in Splunk infrastructure due to its implementation.

Please share if there is anything else that I need to correct or understand with this implementation.

Thank you