topic Re: keep receiving error on two scheduled alerts? in Alerting

keep receiving error on two scheduled alerts?

packet_hunter — Thu, 05 Jan 2017 15:49:42 GMT

I have a DevOps test instance of splunk with some reports (that I run manually ad hoc) and two scheduled alerts.

I keep receiving the following error.

I am not sure I have the schedule alerts setup correctly. I only have two with the following schedules....

Any ideas? Please advise.

Thank you

Re: keep receiving error on two scheduled alerts?

somesoni2 — Thu, 05 Jan 2017 16:13:34 GMT

It seems there are some real-time searches running and exceeding your system's limit. If you've access to the box, check processes (psef splunk) to see what searches are running and get more information based on sid. In fact, the data summary section in the default search page is also a real-time search so check if there multiple windows open.

Re: keep receiving error on two scheduled alerts?

packet_hunter — Thu, 05 Jan 2017 16:18:40 GMT

I will look into it, thank you

Re: keep receiving error on two scheduled alerts?

hunters_splunk — Tue, 29 Sep 2020 12:16:23 GMT

Hi packet_hunter,

You can try increasing your max number of real-time searches by modifying relevant real-time search settings in limits.conf (create limits.conf under /local rather than directly updating the one under /default) .

limits.conf:

base_max_searches =
* A constant to add to the maximum number of searches, computed as a multiplier
of the CPUs.
* Defaults to 6

max_searches_per_cpu =
* The maximum number of concurrent historical searches per CPU. The system-wide
limit of historical searches is computed as:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

max_rt_search_multiplier =
* A number by which the maximum number of historical searches is multiplied to
determine the maximum number of concurrent real-time searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

Hope this helps. Thanks!
Hunter

Re: keep receiving error on two scheduled alerts?

packet_hunter — Mon, 09 Jan 2017 16:16:00 GMT

Thank you for the advice. I will keep this information for future use. But our concern is that we did not have any real-time searches running just the two scheduled searches. Still investigating.

Re: keep receiving error on two scheduled alerts?

packet_hunter — Mon, 09 Jan 2017 16:29:02 GMT

so I did a ps -ef |grep splunk but I am not sure how to find real-time searching.... please provide more advise to find these rogue real-time searches Thank you

Re: keep receiving error on two scheduled alerts?

somesoni2 — Mon, 09 Jan 2017 16:35:25 GMT

It should say search --id=rt_.....
It should give you the user name as well towards the end.

Re: keep receiving error on two scheduled alerts?

packet_hunter — Mon, 09 Jan 2017 16:39:36 GMT

ok thank you

Re: keep receiving error on two scheduled alerts?

packet_hunter — Mon, 09 Jan 2017 18:34:32 GMT

at the moment I cannot find anything with id=rt... but when the message alerts again, I know how to find it. thank you.

Re: keep receiving error on two scheduled alerts?

packet_hunter — Mon, 09 Jan 2017 21:35:01 GMT

fyi, I found the problem, when I kick off the security app, a bunch of real-time searches occurred when the dashboard load.... |grep id=rt revealed the issue. Thanks again.