I have the following message in the scheduler activity window on DMC, that states I have reached the limit of concurrent real-time searches.
08-08-2017 14:33:01.062 +0000 INFO SavedSplunker - savedsearch_id="admin;search;Endpoint Monitoring", search_type="scheduled", user="admin", app="search", savedsearch_name="Endpoint Monitoring", priority=default, status=skipped, reason="The maximum number of concurrent real-time scheduled searches on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1502202780, window_time=0
I am trying to understand where is this limit inherited from, as my limits.conf contains :
You are doing it wrong (or maybe I should say that your sales team sold it to you wrong or your PS team didn't explain it to you right): there is almost never a case where a realtime search is warranted (and "almost" is gross hyperbole).
The first rule is that if you have a human in the response-chain, do not use realtime.
The second rule is that if your (pipeline latency is > your realtime window), which is ALMOST ALWAYS the case, your search is not processing all the events required to be accurate so do not use realtime.
Do yourself and everyone else using your Splunk a HUGE favor. Stop using realtime altogether. Instead, figure out what your pipeline latency is ( | stats max(eval(_indextime - _time)) ) and set your search window ( earliest and latest values ) appropriately and then use a scheduled search periodicity that is somewhere between double and half of your best human response time or your largest acceptable delay. if you can receive the email and respond no more quickly than 10 minutes, then have your search run */10 * * * *.
The only real exception here is DataModel Accelerations (which ITSI uses a ton of). Using realtime is OK there; don't mess with those.
The bottom line is this: for any tool that cares about when each event actually happened, realtime is functionally impossible. Splunk REALLY cares about when an event actually happened, otherwise audits, compliance and forensics would be impossible. If you need a solution that pretend to be realtime (none of them really are), then get one of the other "web-scale" tools that pretends that "when I got the event" is the same thing as "when the event must have actually happened" and use that tool. But don't try to use Splunk for realtime.
P.S. That log is a bug and should be fixed soon. Because realtime searches NEVER END, technically, they are ALWAYS skipped. There is a jira on this; you should just ignore it.