Question Regarding Custom Alert Configuration

[bcarr12]

I currently have the following saved search scheduled to run every 10 minutes:

SearchTerm source="logfile.log" | timechart count as mycount span=1min | eval time=strftime(_time,"%r") | eval mycount=tostring(mycount,"commas") | table time, mycount | rename time AS "Time" | rename mycount AS "My Count"

This generates search results with mycount by minute for a 10 minute interval (something like this):

Time My Count
01:30:00 PM 593
01:31:00 PM 583
01:32:00 PM 537
01:33:00 PM 504
01:34:00 PM 539
01:35:00 PM 622
01:36:00 PM 605
01:37:00 PM 805
01:38:00 PM 694
01:39:00 PM 582

I would like to create a custom condition where an Alert will be generated and emailed out if one of the counts in the results is greater than 1,000. What would be the best way to go about doing this from a custom condition search (or is there a more effective way?)

Thank you!

[sdaniels]

You can create an alert that has the Schedule of 'Monitor in a real-time over a rolling window of...' and set that to 10 mintues. And then put a Trigger if 'Custom condition is met' with search mycount > 1000. Not sure if that will apply to every 1 min bucket in your search though so you'll have to try it.

You might also try adding ... | where mycount > 1000 to the end of your original search and create the alert with the Trigger If Number of Results is greater than 0. IE anything in the list will have a mycount over 1000 and therefore you want to be alerted.

[bcarr12]

Would you be able to provide an example in the context of my search as to how the isint function could be used? I read the doc but I'm not entirely clear on how it works or how I should be using it properly.

[sdaniels]

Yes, you'll need to have another eval for mycount to save off another copy i think. Do a test at the end of your search to make sure mycount is an int. You can use the eval function 'isint(X)'. You'll see an example here. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonEvalFunctions

[bcarr12]

I ended up changing my initial search to:
SearchTerm source="logfile.log" | timechart count as mycount span=1min | eval time=strftime(_time,"%r") | eval mycount_int=tostring(mycount,"commas") | table time, mycount_int | rename time AS "Time" | rename mycount_int AS "My Count"

Then I set the custom condition alert to "search mycount>1" and it is still not returning any results. Do I need another eval for mycount=mycount as well before the change to string? Any other thoughts? Thanks again for all your help thus far, it has been really helpful.

[sdaniels]

I think that should work, yes.

[bcarr12]

Is there any way to "have my cake and eat it too" - for instance use eval to have two variables in my search, one for mycount as a string so it displays nicely in the alert email table, and another for mycountint which would be the integer equivalent value for the conditional alert to check against (but it would not be used in the table)?

[sdaniels]

Just caught that. As a string we're not going to get the right matches. You will want another search with mycount as a int in order for this to work.

[bcarr12]

Hmm, I set it to "search mycount>500" and even though there are some instances over 1,000 it is still not sending an email. Does it matter that I set the value of mycount to a string so a comma is included in results (ie it's no longer an integer)?

[sdaniels]

I think i missed putting search in there. search mycount > 1000

[bcarr12]

Thank you...would you mind being a little more specific regarding the trigger? I have the search scheduled via Cron to run every 10 minutes on weekdays: */10 * * * 1-5, and if I set the custom condition to "mycount>500" (just to test it out), I am getting this error: "Cannot parse alert condition. Search operation 'mycount' is unknown. You might not have permission to run this operation." Is there a certain syntax I should be using?