Most of the answers I've seen to questions like this seem to focus on the idea of reviewing essentially all of your searches and then optimizing all of your searches.  While that may be good practice, it also doesn't necessarily address the actual problem, because it doesn't help you to identify the one or two searches that are specifically causing the health monitor to become red.  Recently when I ran into this issue, I used this search to find the specific offending searches and then fix them, deactivating the alert. index=_internal sourcetype=scheduler "Scheduler Health Report recording a extremely lagged search" ... View more

Cisco provides a recommendation for hardware that should be followed but there are no recommended hardware or architecture setups that will allow the older TA to scale beyond an 8k throughput.  Only an upgrade to the new TA will allow a higher throughput. ... View more

Indeed, Cisco Security (the supporting team) provided additional information that explained in detail what was happening with the TAs involved.  Please see my comments above. ... View more

There is indeed just one big source, because the Cisco FMC concentrates all Cisco logs and then provides the transfer interface via eStreamer. ... View more

An update for those who may have similar issues: Support for these Technical Apps and the eStreamer protocol interface to Splunk is not provided by either Splunk Support or Cisco Support.  Opening tickets with either party did not successfully resolve these questions.  The only support currently available for this interface is to use the email [email protected].  If staff are available to respond, they will answer some limited questions. Cisco Security confirms that Splunkbase #3662 (TA-eStreamer) has been desupported in 2024, because the eNcore client is it based upon has been desupported.  In essence, the eNcore client is being replaced.  As a result, it is recommened (by Cisco Security) that systems with Splunkbase #3662 replace it with the new (supported) TA, which is Splunkbase #7404 (Cisco Security Cloud).   Cisco Security indicates that the older TA-eStreamer had a maximum expected throughput of under 10,000 events per second, and practically limited to under 8K of continuous throughput.  So, applications like our that routinely exceed 8K per second would never have successfully used TA-eStreamer for that performance level.  Performance levels for the newer Cisco Security Cloud app (#7404) are expected to be in the maximum range of 15-20K events per second, because it uses a new eStreamer SDK that replaces to decommissioned eNcore client software.  Since it is possible our application may rise above that level, I asked if the app potentially supported using things like load balancers to scale beyond 15-20K, but no answer was provided for this question. The Cisco Security team responding to my questions indicated that support for this TA is somewhat limited, with only a best effort support during Eastern US office hours. The Cisco Security team of course also recommends using hardware that follows their documented guidelines and provides sufficient memory, disk, and CPU to run the software at its maximum performance levels.  But, once those maximum performance levels are reached (15-20K per second) there is no recommended scaling beyond that.   Our team expects to experiment in various setups and architectures to see how far we can push the new TA. ... View more

I've read the documentation for how to accept input for things like "Interval" (typically in seconds) which I think is your point #1.  That explains how to get the settings configured, but not how to actually achieve the scheduled interval when running your API calls. As for alerts and scheduled reports - it seems like what you are saying is one approach is to write some code that will generate custom SPL functions that will call your API, then use the normal search scheduler to make that custom function run on a regular basis.  That sounds oddly difficult - is that a common approach that technical app developers use for scheduling their API calls? ... View more

This is a really old post but I had the same problem.  A search query that appears to be helping me find these problems is: index=_internal sourcetype=splunkd log_level=ERROR component=HttpInputDataHandler The results are imperfect because they don't exactly match what's shown in the authentication failures, but in my case, it appears the errors are being caused by a source that is sending in blank/missing tokens. ... View more

This ticket has been open a long time, but just in case someone else runs into it... Some people include workload  specifications with the URL of a search they've done. When  this occurs, and they share the link with others, then they can get this  error if the other person does not have the right granted to them to select a workload pool. This most often happens when a Splunk admin shares  a link with a non-admin.  The fix is to remove any reference to a workload pool  from the URL.  Or, alternately, you can grant the role of the person running the search to allow them to list  and select workload pools.  But if you do the second one, then anyone with that role can select which workload pool they run in. ... View more