Optimizing the expensive saved searches

[tech_g706]

Hi,

I am seeking recommendations on optimizing the most resource-intensive saved searches in my Splunk Cloud instance to reduce Indexers CPU utilization, which is consistently at 99%. We are using Splunk ES, SA-NetworkProtection apps. 

By CMC, these are the most expensive ones and take around 30-40 minutes to complete. 

_ACCELERATE_DM_Splunk_SA_CIM_Authentication_ACCELERATE_
_ACCELERATE_DM_Splunk_SA_CIM_Network_Traffic_ACCELERATE_
_ACCELERATE_DM_Splunk_SA_CIM_Vulnerabilities_ACCELERATE_
_ACCELERATE_DM_Splunk_SA_CIM_Endpoint.Services_ACCELERATE
_ACCELERATE_DM_Splunk_SA_CIM_Network_Sessions_ACCELERATE_
_ACCELERATE_DM_Splunk_SA_CIM_Change_ACCELERATE_
_ACCELERATE_DM_SA-NetworkProtection_Domain_Analysis_ACCELERATE_
_ACCELERATE_DM_DA-ESS-ThreatIntelligence_Threat_Intelligence_ACCELERATE_


Any recommendations on how I can optimize without disabling them?

Thank you

[bowesmana]

Improving the DM acceleration searches can be tricky, as others have pointed out, so can you identify other non DMA searches that are at the top of the list. Searches that just take a long time are not necessarily the bad searches, they may just be handling large datasets.

Poor performing searches can come from badly written dashboard searches that use joins or other poor techniques. They can also come from bad saved searches, again due to bad search techniques.

It's often these user written searches that can bring Splunk to its knees. Of course it's also possible that you just don't have enough grunt - what licence model is your Splunk Cloud using, SVCs or ingest?

[tech_g706]

@bowesmana  Thanks for the response.

We are using SVCs model. We have Splunk ES and other resource intensive apps.
I am looking for a way to optimize searches as the current indexer memory utilization is 99% since last 2 months. 
I used the CMC dashboard and that showing me the DM acceleration searches are the most expensive ones.

[livehybrid]

Hi @tech_g706 

Those searches are accelerating the Data Models, presumably you are using Splunk Enterprise security? I think the first thing to check is are you actually using all of those models for your ES rules/searches?

Secondly I would check that you have set the specific required indexes in the allowed index list for each of your Data Models in the "CIM Setup" section of ES, by default these are set to index=* but should be configured to only access the indexes that contain the relevant data for the particular data model. 

Check out these docs for more information on managing data models in ES.

The last thing I would check is the Data Model audit dashboard (Audit > Data Model Audit) in ES, this should give you some stats on how the DM are behaving and if they are updating correctly. 

You can also check out https://docs.splunk.com/Documentation/ES/8.0.2/Install/ConfigureDatamodels#Data_model_acceleration_r... which has some further details on the configuration options such as the summary period for each data model.

🌟 Did this answer help you? If so, please consider:

• Adding kudos to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing.

[gcusello]

Hi @tech_g706 ,

it isn't possible to optimize accelerated scheduled searches, you can only reduce the execution frequency, if this is compatible with your requisites.

E.G. if you schedule acceleration searches every 10 or 15 minutes, instead of 5, you will have avalible data later than now, so you must change the execution time window of your Correlation Searches.

In other words, if having a frequency of 5 minutes, you can use a time period from -10m@m to -5m@m  , having a frequency of 15 minutes, you must schedule Correlation Searches from -20m@m to -15m@m, is this acceptable for you?

Otherwise, you have to use summariesonly=false, but you lose in performances.

Ciao.

Giuseppe