SplunkEnterpriseSecurity App on ClusterMaster

[raiqb01]

Hi,
While troubleshooting below error message: 

"The percentage of non high priority searches delayed (75%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=16. Total delayed Searches=12"

how can I address actual issue?

=============

while looking into the system, I found out that 

1- Splunk ES app is installed under /opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite.

Can I remove the app from above location?

2- furthermore, The output of below query is :
index=_internal sourcetype=scheduler savedsearch_name=* status=skipped
| stats count BY reason

1- Error in 'SearchParser': The search specifies a macro 'notable' that cannot be found. Reasons include: the macro name is misspelled, you do not have...

2-The maximum number of concurrent running jobs for this historical scheduled search on this instance has been reached

=================

I found that 

[gcusello]

Hi @raiqb01 ,

sorry but you are doing a little of confusion:

ES isn't to install on Indexers and Cluster Manager, ES must be installed only on Search Head and (if you have a Search Head Cluster) on Deployer.

On the Indexers  (using Cluster Manager) you must install an add-on that must be downloaded from ES installation on Search Heads.

The issue that you're reporting isn't related to ES, but you should analyze your inputs add-on because the issue is probably related to the fact that you don's correctly assign the sourcetype to your logs.

Look at the Linux Add-On if present.

Then the Error 1 is related to a nn correct ES installation.

Last: the Issue2 is related to few resources for your Indexers and Search Heads.

Didì you followed the ES installation instructions (https://docs.splunk.com/Documentation/ES/7.3.2/Install/InstallEnterpriseSecurity) ?

what's the reference hardware that you're using for ES (https://docs.splunk.com/Documentation/ES/7.3.2/Install/DeploymentPlanning ) ?

Ciao.

Giuseppe