Alerting

keep receiving error on two scheduled alerts?

packet_hunter
Contributor

I have a DevOps test instance of splunk with some reports (that I run manually ad hoc) and two scheduled alerts.

I keep receiving the following error.

alt text

I am not sure I have the schedule alerts setup correctly. I only have two with the following schedules....

alt text

Any ideas? Please advise.

Thank you

Tags (2)
0 Karma
1 Solution

hunters_splunk
Splunk Employee
Splunk Employee

Hi packet_hunter,

You can try increasing your max number of real-time searches by modifying relevant real-time search settings in limits.conf (create limits.conf under /local rather than directly updating the one under /default) .

limits.conf:

base_max_searches =
* A constant to add to the maximum number of searches, computed as a multiplier
of the CPUs.
* Defaults to 6

max_searches_per_cpu =
* The maximum number of concurrent historical searches per CPU. The system-wide
limit of historical searches is computed as:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

max_rt_search_multiplier =
* A number by which the maximum number of historical searches is multiplied to
determine the maximum number of concurrent real-time searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

Hope this helps. Thanks!
Hunter

View solution in original post

0 Karma

hunters_splunk
Splunk Employee
Splunk Employee

Hi packet_hunter,

You can try increasing your max number of real-time searches by modifying relevant real-time search settings in limits.conf (create limits.conf under /local rather than directly updating the one under /default) .

limits.conf:

base_max_searches =
* A constant to add to the maximum number of searches, computed as a multiplier
of the CPUs.
* Defaults to 6

max_searches_per_cpu =
* The maximum number of concurrent historical searches per CPU. The system-wide
limit of historical searches is computed as:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

max_rt_search_multiplier =
* A number by which the maximum number of historical searches is multiplied to
determine the maximum number of concurrent real-time searches
* Note: the maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
* Defaults to 1

Hope this helps. Thanks!
Hunter

0 Karma

packet_hunter
Contributor

Thank you for the advice. I will keep this information for future use. But our concern is that we did not have any real-time searches running just the two scheduled searches. Still investigating.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It seems there are some real-time searches running and exceeding your system's limit. If you've access to the box, check processes (psef splunk) to see what searches are running and get more information based on sid. In fact, the data summary section in the default search page is also a real-time search so check if there multiple windows open.

packet_hunter
Contributor

so I did a ps -ef |grep splunk but I am not sure how to find real-time searching.... please provide more advise to find these rogue real-time searches Thank you

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It should say search --id=rt_.....
It should give you the user name as well towards the end.

0 Karma

packet_hunter
Contributor

fyi, I found the problem, when I kick off the security app, a bunch of real-time searches occurred when the dashboard load.... |grep id=rt revealed the issue. Thanks again.

0 Karma

packet_hunter
Contributor

at the moment I cannot find anything with id=rt... but when the message alerts again, I know how to find it. thank you.

0 Karma

packet_hunter
Contributor

ok thank you

0 Karma

packet_hunter
Contributor

I will look into it, thank you

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...