Splunk Search

Alert from 2 searched values that calculate a percentage

sberg
Explorer

First time posting! --using splunk 4.2.4--

I noticed similar questions on here that were either unanswered or didn't quite meet my needs. I had quite the time getting this far so thought I'd share and also solicit feedback that could improve on this. I'll also note that the logs I'm working with are really messy and inconsistent.

There are really 3 parts:
1) get 2 unique values from a search so you can do some math with them
2) in that same search, calculate a percentage from these 2 values
3) using this search trigger an alert with a 3rd condition

For my needs, a static number of errors to alert on just wouldn't cut it due to large fluctuations in the number of users. Using a ratio of errors/users (in the last X hrs) was preferable.

sourcetype="myAppHosts" "SOME SPECIFIC ERROR EVENT STRING" 
| stats count as error_total
| eval user_total = [search "SOME UNIQUE USER LOGIN EVENT STRING" | stats count as search]
| eval percent = (error_total/user_total)*100 
| fields percent error_total user_total

We then create an alert validating a custom condition against the calculated percentage...

search percent > 10

The resulting email provides the 3 fields in the last part of the search.

This could be modified to use some regex in place of string searches (to dedup users,etc..), but it works for my needs as is. If there is a better way to do this, I'm all ears.

Cheers.

0 Karma
1 Solution

sberg
Explorer

Figured it out... 🙂 Thanks again.

sourcetype="myApp" ("ERROR String" NOT "Some string to exclude an error type") OR
("Count logins string 1" OR "Count logins string 2")
| stats count(eval(searchmatch("ERROR String"))) as errors,
count(eval(searchmatch("Count logins string 1") OR searchmatch("Count logins string 2"))) as logins
| eval percentage=100*errors/logins
| fields percentage errors logins

View solution in original post

0 Karma

sberg
Explorer

Figured it out... 🙂 Thanks again.

sourcetype="myApp" ("ERROR String" NOT "Some string to exclude an error type") OR
("Count logins string 1" OR "Count logins string 2")
| stats count(eval(searchmatch("ERROR String"))) as errors,
count(eval(searchmatch("Count logins string 1") OR searchmatch("Count logins string 2"))) as logins
| eval percentage=100*errors/logins
| fields percentage errors logins

View solution in original post

0 Karma

sberg
Explorer

Ahhh... makes sense.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

actually you can do any base search expression in searchmatch(), e.g., you can rewrite as: ... | stats count(eval(searchmatch("\"ERROR String\" NOT \"Some string to exclude an error type\""))) as errors, count(eval(searchmatch("\"Count logins string 1\" OR \"Count logins string 2\""))) | ... as long as you escape the quotes in the search strings.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

This would work a bit better in most cases:

("error string") OR ("login event string")
| stats count(eval(eval_expression_to_identify_errs)) as errs, count(eval(eval_expression_to_identify_logins)) as logins
| eval pct=100*errs/logins

The eval expressions might be as simple as result_code==999 or operation=="login", or you might have to use the match() or searchmatch() eval functions.

0 Karma

sberg
Explorer

Thanks for the reply, and for another option. The logs I'm searching in don't follow the key=value format. They're all over the place depending on who coded it, and often the vital info is just dumped out into a string of readable text.

As you suggested, this works well with searchmatch(), but only if the expression is simple. What about when there are multiple conditions? I'm not sure how to express this in searchmatch().

example:
ERROR NOT "apply delta" NOT "Could Not Get File" NOT "Couldn't open file"

Thanks!

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!