We have a process that produces 8,000 requests per second that are consumed by a server. We average only about 2 timeout events per second. A few times per day, however, this timeout rate will spike to 1,000 timeouts for no more than a second or two. We don't care about these spikes, but need to know as quickly as possible when the consuming service is down. How can I "smooth" the event count such that we can ignore the spikes and be notified within about 5 minutes of an outage? I'm thinking of something like a kalman filter (I'm not a mathematicition) acting on the past 5 minutes of data and runs every 5 minutes. A normal average won't do the trick because it can't tell the difference between performance degradation and a spike. It doesn't seem like the predictive functions native to Splunk would work right out of the box. Any other ideas? Thanks. ... View more

Several of us here in my office have a lot of trouble logging into Splunk. After entering my username and password the Splunk login web page gives the error: "Invalid username or password.". It is then possible to simply refresh the browser or type our Splunk URL directly into the browser (as opposed to using a previously bookmarked link to a search) to bypass the error. Looking at the splunkd.log I see Splunk spit out this error when my login failure occurs: 08-08-2013 15:37:34.741 +0000 ERROR ScriptRunner - stderr from '': PHP Notice: Undefined index: roles in [path]/splunk/scripts/ssoScript.php on line 84 08-08-2013 15:37:34.742 +0000 WARN AuthenticationManagerScripted - Function 'userLogin' failed. Could not find '--status=success' in output 08-08-2013 15:37:34.742 +0000 ERROR AuthenticationManagerScripted - Script function userLogin failed for user: [userid] Any assistance would be greatly appreciated as this is rather annoying. Thanks. ... View more