How to resolve skipped scheduled searches?

[Hemnaath]

Hi All, Currently we could see few skipped schedule searches in our environment. From the deployment instance we have ran this query to get the list of skipped searches.

Query details:

dmc_set_index_internal sourcetype=scheduler (status="skipped")

Event details:

08-24-2017 03:12:12.981 -0400 INFO  SavedSplunker - savedsearch_id="nobody;DA-deployment_monitor;_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_DA-deployment_monitor_nobody_1a56f43bf8d5bf20_ACCELERATE_", search_type="report_acceleration", user="nobody", app="DA-deployment_monitor", savedsearch_name="_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_DA-deployment_monitor_nobody_1a56f43bf8d5bf20_ACCELERATE_", priority=default, status=skipped, reason="The maximum number of concurrent auto-summarization searches on this instance has been reached", concurrency_category="summarization_scheduled", concurrency_context="instance-wide", concurrency_limit=2, scheduled_time=1503558600, window_time=0

APP Name : DA-deployment_monitor
hostname: test01

Event details:

08-24-2017 03:00:13.752 -0400 INFO  SavedSplunker - savedsearch_id="splunk;sos;_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_sos_splunk_9fd0bac7cd608f2c_ACCELERATE_", search_type="report_acceleration", user="splunk", app="sos", savedsearch_name="_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_sos_splunk_9fd0bac7cd608f2c_ACCELERATE_",: priority=default, status=skipped, reason="The maximum number of concurrent historical scheduled searches on this instance has been reached", ", concurrency_context="instance-wide", concurrency_limit=4, scheduled_time=1503558000, window_time=0

App name: sos
hostname:test01

Event details:

08-24-2017 03:23:02.291 -0400 INFO  SavedSplunker - savedsearch_id="nobody;search;_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_search_nobody_365ca83246f2cca8_ACCELERATE_", search_type="report_acceleration", user="nobody", app="search", savedsearch_name="_ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_search_nobody_365ca83246f2cca8_ACCELERATE_", priority=default, status=skipped, reason="The maximum number of concurrent auto-summarization searches on this instance has been reached", concurrency_category="summarization_scheduled", concurrency_context="instance-wide", concurrency_limit=2, scheduled_time=1503559380, window_time=0

App name: search
hostname:test01

All these events are generated from the Deployment instance "test01"

Question :

1) Unable to find out the exact saved search name "ACCELERATE_D4D707D0-38F3-4F47-A1AA-9DD305E110D0_DA-deployment_monitor_nobody_1a56f43bf8d5bf20_ACCELERATE" from the events , so how to find that out inside the app directory

/opt/splunk/etc/apps/DA-deployment_monitor/default/savedsearches/ -- I could see many queries saved as scheduled searches, but could not find the exact name mentioned as mentioned in the above events.

2) How to fix this issue, I could see these reasons from events

a) "The maximum number of concurrent auto-summarization searches on this instance has been reached"
b) concurrency_category="historical_scheduled
c) concurrency_category="summarization_scheduled"
d) concurrency_limit=2
e) scheduled_time=1503558600
f) window_time=0
g) concurrency_context="instance-wide"

Kindly guide me how to fix this issue.

thanks in advance.

[cmerriman]

it isn't recommended that you manually edit or manage data models in the data model files. you should do as much as you can in Splunk Web.

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Managedatamodels#Manual_data_model_mana...

to edit a data model, you can follow this procedure:
1. http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Managedatamodels#Navigating_to_the_Data...
2. http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Managedatamodels#Enable_data_model_acce...
however, i prefer to get to Settings (from the Splunk Bar)>Data models to see all of the data models vs the Data sets tab. You can then edit the acceleration. You can pick a smaller acceleration summary range, or turn acceleration off. You aren't allowed to edit the data model while acceleration is turned on. If you wish to edit the data model to perhaps make the search more efficient, turn off acceleration, edit the data model, and then turn it back on.

[Hemnaath]

Hi Cmerriman, thanks for you effort on this, I had checked in the test01 deployment console -->settings-->datamodel -- but unable to find any data model configured in this instance. So can you please tell me how to find the which data model is configured for the apps/saved search causing the issue.

App name: search
APP Name : DA-deployment_monitor
App name: sos

Kindly help me on this.

[gjanders]

The reason ""The maximum number of concurrent auto-summarization searches on this instance has been reached" is advising that the currently running summarization searches have not completed and the scheduler cannot start the next summarization search.

In other words you have an accelerated data model that is running a search that cannot complete during the required timeframe, in this case it appears to be coming from the deployment monitor.

Can you check for accelerated data models on the server and the timeframe? Clearly the query that is been run cannot complete quickly enough and that is why you are seeing this error...

[Hemnaath]

Hi Garethatiag, thanks for your effort on this, Can you please guide me how/where I can find the accelerated data models setting on test01 which is the deployment server. From my DMC console, after executing the above query, I could see the above events and from that events, I can fetch the app name and server name not the exact savedsearches name.

Question :

In test01 the below apps are configured but not sure how/where to find the accelerated data models.conf file and saved search details. And also how to fix this issue.

App name: search
APP Name : DA-deployment_monitor
App name: sos

Please garethatiag, help me on the above question, as this issues is pending for more than a week.

[Hemnaath]

Hi Garethatiag, thanks for your effort on this, test01 is the deployment instance and all the three apps are configured under instance, but when checked the deployment console -->settings--knowledge -->datamodel unable to find the data model in deployment instance

But I could see the data model being configured in the cluster search head master server instance. But unable to locate these app in that server under--> opt/splunk/etc/apps/. So can you please help me in how to find out which data model is configured for the apps/saved search causing the issue.

App name: search
APP Name : DA-deployment_monitor
App name: sos

This is the first time I am facing this issue, not having much idea on data models, so please do help
me.

[gjanders]

As per the documentation for datamodels can you confirm your inspecting the data model settings in the GUI on a search head...(not the deployer server)

[Hemnaath]

Hi Garethatiag, thanks for your effort on this issue. There are 6 apps throwing the search scheduler status = skipped, out which three apps are configured in the cluster master search heads. search head console -->settings--knowledge -->datamodels.

Query details:
dmc_set_index_internal sourcetype=scheduler (status="skipped")

Event details:

08-27-2017 23:22:51.627 -0400 INFO SavedSplunker - savedsearch_id="nobody;critical_security_controls;ACCELERATE_C090FDA2-105E-4875-A110-3F13FF986151_critical_security_controls_admin_4b2771dc07d5983d_ACCELERATE", search_type="report_acceleration", user="admin", app="critical_security_controls", savedsearch_name="ACCELERATE_C090FDA2-105E-4875-A110-3F13FF986151_critical_security_controls_admin_4b2771dc07d5983d_ACCELERATE", priority=default, status=skipped, reason="The maximum number of concurrent auto-summarization searches on this cluster has been reached", concurrency_category="summarization_scheduled", concurrency_context="cluster-wide", concurrency_limit=15, scheduled_time=1503890400, window_time=0

App name : critical_security_controls

There are 23 data models configured on this app.
1) Alerts 2) Application state 3) authentication 4)certificates 5) change analysis, 6) Data Loss Prevention 7) CIM Validation (S.O.S) 😎 Database 9) Email 10) Hybris script failure 11) Interprocess messaging 12) Instrution detection 13)Inventory 14)JVM 15)Malware 16)Network Resolution (DNS) 17) Network session 18) Network Traffic 19) Performance 20 )splunk audit logs 21) Splunk's internal Audit logs -Sample 22) Splunk's Internal Server Audit logs-Sample 23) Ticket Management 24) Web 25)Vulnerbilities 26)Updates

08-27-2017 23:33:07.111 -0400 INFO SavedSplunker - savedsearch_id="nobody;Splunk_TA_CIM;ACCELERATE_DM_Splunk_TA_CIM_Authentication_ACCELERATE", search_type="datamodel_acceleration", user="nobody", app="Splunk_TA_CIM", savedsearch_name="ACCELERATE_DM_Splunk_TA_CIM_Authentication_ACCELERATE", priority=highest, status=skipped, reason="The maximum number of concurrent auto-summarization searches on this cluster has been reached", concurrency_category="summarization_scheduled", concurrency_context="cluster-wide", concurrency_limit=15, scheduled_time=1503891180, window_time=0

App name: Splunk_TA_CIM

08-27-2017 20:35:05.695 -0400 INFO SavedSplunker - savedsearch_id="nobody;symantec_app;ACCELERATE_DM_symantec_app_Symantec_Endpoint_ACCELERATE", search_type="datamodel_acceleration", user="nobody", app="symantec_app", savedsearch_name="ACCELERATE_DM_symantec_app_Symantec_Endpoint_ACCELERATE", priority=default, status=skipped, reason="The maximum number of concurrent auto-summarization searches on this cluster has been reached", concurrency_category="summarization_scheduled", concurrency_context="cluster-wide", concurrency_limit=15, scheduled_time=1503880500,

App name : symantec_app

Kindly guide me how to fix this problem.

[Hemnaath]

Hi Garethatiag, I have captured everything from the search head console--settings-datamodels-app in the above comments. Kindly guide me to fix the issue.

thanks in advance.

[gjanders]

As per https://answers.splunk.com/answers/400227/maximum-number-of-historical-concurrent-system-wid.html you could increase the number of concurrent searches your server can run, this will add more load to your indexers.

If your indexers are not overly busy with CPU/IO then increasing the number of concurrent searches may be a valid option for you, please refer to the linked answer or the limits.conf for configuration you can change.

Changing this setting will require a restart in version 6.5.x and below (I assume it will in higher versions such as 6.6.x but I have not checked)

[Hemnaath]

Hi Garethatiag, Good Morning, hey need to know how to take the list of all auto-summarization searches from search head cluster.

Could you please guide me on how / where I can get this list .
thanks in advance.

[gjanders]

I do not have a search for this, i suspect you could query the REST API for all searches and then search for the summarise command or similar.
There might be an answer on SplunkAnswers already or you might want to create a new question.

[Hemnaath]

Hi All can anyone guide on this issue.