Alerting

Question Regarding Custom Alert Configuration

Path Finder

I currently have the following saved search scheduled to run every 10 minutes:

SearchTerm source="logfile.log" | timechart count as mycount span=1min | eval time=strftime(_time,"%r") | eval mycount=tostring(mycount,"commas") | table time, mycount | rename time AS "Time" | rename mycount AS "My Count"

This generates search results with mycount by minute for a 10 minute interval (something like this):

Time My Count
01:30:00 PM 593
01:31:00 PM 583
01:32:00 PM 537
01:33:00 PM 504
01:34:00 PM 539
01:35:00 PM 622
01:36:00 PM 605
01:37:00 PM 805
01:38:00 PM 694
01:39:00 PM 582

I would like to create a custom condition where an Alert will be generated and emailed out if one of the counts in the results is greater than 1,000. What would be the best way to go about doing this from a custom condition search (or is there a more effective way?)

Thank you!

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

You can create an alert that has the Schedule of 'Monitor in a real-time over a rolling window of...' and set that to 10 mintues. And then put a Trigger if 'Custom condition is met' with search mycount > 1000. Not sure if that will apply to every 1 min bucket in your search though so you'll have to try it.

You might also try adding ... | where mycount > 1000 to the end of your original search and create the alert with the Trigger If Number of Results is greater than 0. IE anything in the list will have a mycount over 1000 and therefore you want to be alerted.

0 Karma

Path Finder

Would you be able to provide an example in the context of my search as to how the isint function could be used? I read the doc but I'm not entirely clear on how it works or how I should be using it properly.

0 Karma

Splunk Employee
Splunk Employee

Yes, you'll need to have another eval for mycount to save off another copy i think. Do a test at the end of your search to make sure mycount is an int. You can use the eval function 'isint(X)'. You'll see an example here. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonEvalFunctions

0 Karma

Path Finder

I ended up changing my initial search to:
SearchTerm source="logfile.log" | timechart count as mycount span=1min | eval time=strftime(time,"%r") | eval mycountint=tostring(mycount,"commas") | table time, mycountint | rename time AS "Time" | rename mycountint AS "My Count"

Then I set the custom condition alert to "search mycount>1" and it is still not returning any results. Do I need another eval for mycount=mycount as well before the change to string? Any other thoughts? Thanks again for all your help thus far, it has been really helpful.

0 Karma

Splunk Employee
Splunk Employee

I think that should work, yes.

0 Karma

Path Finder

Is there any way to "have my cake and eat it too" - for instance use eval to have two variables in my search, one for mycount as a string so it displays nicely in the alert email table, and another for mycountint which would be the integer equivalent value for the conditional alert to check against (but it would not be used in the table)?

0 Karma

Splunk Employee
Splunk Employee

Just caught that. As a string we're not going to get the right matches. You will want another search with mycount as a int in order for this to work.

0 Karma

Path Finder

Hmm, I set it to "search mycount>500" and even though there are some instances over 1,000 it is still not sending an email. Does it matter that I set the value of mycount to a string so a comma is included in results (ie it's no longer an integer)?

0 Karma

Splunk Employee
Splunk Employee

I think i missed putting search in there. search mycount > 1000

0 Karma

Path Finder

Thank you...would you mind being a little more specific regarding the trigger? I have the search scheduled via Cron to run every 10 minutes on weekdays: */10 * * * 1-5, and if I set the custom condition to "mycount>500" (just to test it out), I am getting this error: "Cannot parse alert condition. Search operation 'mycount' is unknown. You might not have permission to run this operation." Is there a certain syntax I should be using?

0 Karma