I have not activated anything other than the correlation events and queries that splunk enterprise security brings by default, I just set them from inactive to active.
This may be your first mistake. Enabling all of the built-in searches in ES is asking for trouble. Not all ES correlation searches apply to all environments so it's essential to review each CS to determine if it should be enabled.
What is the possible cause?
The cause is given in the message. You're running too many searches at a time.
Where should I check?
Start with the Monitoring Console. Select Search->Scheduler Activity: Instance then scroll down to the "Count of Skipped Reports by Name and Reason" panel. This will list the most commonly skipped searches.
Take a look at the "Count of Skipped Reports Over Time" graph. It will show the time of day when searches are skipped. Make note of the high and low periods so you can change schedules from peak times to non-peak times.
If I have to modify the limits.conf file what is the path of that file?
You probably will not have to modify limits.conf, however, there are several places where the file could be. The most likely place is $SPLUNK_HOME/etc/system/local, however, config files can be in any app directory. You can use btool to find the config file that contains the setting you want to change, but the setting can be overridden by any app with greater precedence (see https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Wheretofindtheconfigurationfiles)
splunk btool --debug limits list | grep -i "<setting>"
What are the values that I should add or modify from that file?
You shouldn't need to modify anything in limits.conf. Try to reduce the number of searches running at any given time before resorting to increasing the run limit.
Disable any search or datamodel acceleration that is not used or has no data. For the rest of the searches, set the Schedule Window to "auto". This will let Splunk automatically adjust the start time of a search instead of skipping it.
Review the schedule of each search. To see them in one place, run this search:
| rest splunk_server=local /servicesNS/-/-/saved/searches
| search is_scheduled=1 disabled=0
| fields dispatch.earliest_time dispatch.latest_time eai:acl.owner eai:acl.sharing search cron_schedule title
| rename dispatch.earliest_time as earliest_time, dispatch.latest_time as latest_time, eai:acl.owner as owner, eai:acl.sharing as sharing
| table title cron_schedule earliest_time latest_time schedule_window owner sharing
Look for many searches starting at the same time. It's very common, for instance, for most searches to start at the top of the hour, but it's not always necessary to do so. For each search that starts at the same time, adjust the schedule so the searches are spread out across time.
Next, look at the frequency of each search. Searches that run too often just steal resources from other searches that need to run.
Finally, try to reduce the run-time of your searches. The sooner a search completes the sooner a CPU becomes available and the less likely another search will have to be skipped. In the MC, select Search->Search Activity: Instance and scroll down to "Aggregate Search Runtime". Select "Search name" from the drop-menu to see your longest running searches. Examine each to see if they can be modified to run faster.
---
If this reply helps you, Karma would be appreciated.
