Alerting
Highlighted

keep receiving error on two scheduled alerts?

Contributor

I have a DevOps test instance of splunk with some reports (that I run manually ad hoc) and two scheduled alerts.

I keep receiving the following error.

alt text

I am not sure I have the schedule alerts setup correctly. I only have two with the following schedules....

alt text

Any ideas? Please advise.

Thank you

Tags (2)
0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

SplunkTrust
SplunkTrust

It seems there are some real-time searches running and exceeding your system's limit. If you've access to the box, check processes (psef splunk) to see what searches are running and get more information based on sid. In fact, the data summary section in the default search page is also a real-time search so check if there multiple windows open.

Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

I will look into it, thank you

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

so I did a ps -ef |grep splunk but I am not sure how to find real-time searching.... please provide more advise to find these rogue real-time searches Thank you

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

SplunkTrust
SplunkTrust

It should say search --id=rt_.....
It should give you the user name as well towards the end.

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

ok thank you

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

at the moment I cannot find anything with id=rt... but when the message alerts again, I know how to find it. thank you.

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

fyi, I found the problem, when I kick off the security app, a bunch of real-time searches occurred when the dashboard load.... |grep id=rt revealed the issue. Thanks again.

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Splunk Employee
Splunk Employee

Hi packet_hunter,

You can try increasing your max number of real-time searches by modifying relevant real-time search settings in limits.conf (create limits.conf under /local rather than directly updating the one under /default) .

limits.conf:

basemaxsearches =
* A constant to add to the maximum number of searches, computed as a multiplier
of the CPUs.
* Defaults to 6

maxsearchespercpu =
* The maximum number of concurrent historical searches per CPU. The system-wide
limit of historical searches is computed as:
max
histsearches = maxsearchespercpu x numberofcpus + basemaxsearches
* Note: the maximum number of real-time searches is computed as:
maxrtsearches = maxrtsearchmultiplier x maxhist_searches
* Defaults to 1

maxrtsearchmultiplier =
* A number by which the maximum number of historical searches is multiplied to
determine the maximum number of concurrent real-time searches
* Note: the maximum number of real-time searches is computed as:
max
rtsearches = maxrtsearchmultiplier x maxhistsearches
* Defaults to 1

Hope this helps. Thanks!
Hunter

View solution in original post

0 Karma
Highlighted

Re: keep receiving error on two scheduled alerts?

Contributor

Thank you for the advice. I will keep this information for future use. But our concern is that we did not have any real-time searches running just the two scheduled searches. Still investigating.

0 Karma