topic Re: Custom Conditions in Splunk Search

Custom Conditions

exd42062 — Wed, 09 Oct 2013 16:02:38 GMT

I am new to splunk and want to create an alert for switch ports that flap. So, I need a way to alert if a port on a given switch goes up and down more than a certain number of times per minute. How can I setup the search/alert for that? Below is an example of the search results.

Oct  9 12:01:18 hos-a-3550-1.rockefeller.internal 2635634: Oct  9 12:01:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down

Re: Custom Conditions

lukejadamec — Wed, 09 Oct 2013 20:00:06 GMT

Are there any fields in this event, or is this string the contents of a single field?

Re: Custom Conditions

exd42062 — Thu, 10 Oct 2013 14:10:01 GMT

single field

Re: Custom Conditions

lukejadamec — Thu, 10 Oct 2013 14:22:21 GMT

That should be pretty easy then. You'll need to use rex to extract the FastEthernet0/3 and the 'changed to down/up' information as fields in the search, then you can run statistics on them, and alert when the number for an interface is higher than you'd like.

Try to create the extractions with Field Extractor (down arrow key next to the event). It is easy to use and works most of the time.

Re: Custom Conditions

exd42062 — Fri, 11 Oct 2013 12:45:09 GMT

I created a field extraction for FastEthernet0/3 and when I test it gives me the right list of values. I also created an even type for the Up/Down events. My problem is I don't know what to do next. Do I need to extract the Etehrnet field and the Up/Down together in one field or separate fields. And then I'm not sure what stats to run. I've read about count and distinct count ,but I'm not sure how to formulate the query from there.

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 14:05:01 GMT

We can help with that. I'm not the best with this sort of search, but a lot of folks are.
What did you name the fields?

Re: Custom Conditions

exd42062 — Fri, 11 Oct 2013 14:12:10 GMT

In Field Extractions I've got "interface" and 'up-down".

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 14:17:20 GMT

Try this and see if it gives you the results you want.

 index=yourindex sourcetype=yoursourcetype interface="*" up-down="*" |stats count by interface,up-down | where count>1000 |table interface,up-down,count

Change the 1000 to what ever value you want.

From the time picker on the right, select real time 1 minute window.
From the Create dropdown on the right, select Alert.

If you want to extract the fields in the search then you can use rex:

index=yourindex sourcetype=yoursourcetype | rex "interface (?<Interface>[^, ]*), changed state to (?<state>\w+)" |stats count by Interface,state | where count>1000| table Interface state

Re: Custom Conditions

exd42062 — Fri, 11 Oct 2013 15:13:57 GMT

Am i supposed to put something in the quotes? I get no results.

I could have any number of different interfaces go up and down and I want to see the ones that go up and down a lot. I don't care about the ones that just up and down once or twice.

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 15:17:17 GMT

Sorry, the * were stripped out. I'll fix it.

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 15:26:29 GMT

The way this search works is it works with your log event triggers. For any particular interface you will only get this event when the state changes from up to down or down to up, so for any particular interface over the course of a minute you get >x number of state changes it will show up in the table. You decide what x should be.

Set the number to 0 for testing.

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 15:28:39 GMT

I'm heading offline for about 8 hours. I know this works because I tested on windows EventCodes,ComputerName log entries per minute. Good luck.

Re: Custom Conditions

twinspop — Fri, 11 Oct 2013 15:32:32 GMT

If you're going to search over a time range and want to know if you were flapping during any 1 minute in the time range:

<your search> | bucket span=1m _time | stats count by interface host _time | where count>1000

If you just want to check the last minute:

<your search> earliest=-1m@m latest=@m | stats count by interface host | where count>1000

Re: Custom Conditions

exd42062 — Fri, 11 Oct 2013 15:33:07 GMT

While I wating, I tried the * and that worked for the "interface" field I created, but not for the "up-down" field. Not sure the up-down field is working right.

Re: Custom Conditions

lukejadamec — Fri, 11 Oct 2013 15:38:10 GMT

The up-down field should be included in the field list to the right. The field name will be case sensitive.
Also, for testing you should search a large enough timespan to ensure you are getting some events with up-down values.

Re: Custom Conditions

exd42062 — Fri, 11 Oct 2013 19:09:51 GMT

I am typing the field name right...it does not show up. The time frame is plenty, since i'm seeing these literally every second. I tested the extraction while creating it and it worked so I'm confused why it won't work with a search. The interface extraction works fine.

Re: Custom Conditions

lukejadamec — Sat, 12 Oct 2013 00:47:13 GMT

Perhaps you need an inline field extraction.

Re: Custom Conditions

mikaelbje — Sat, 12 Oct 2013 11:07:05 GMT

Why don't you try the Cisco IOS app out? All the extractions are there. You just need to set up the alert.

Re: Custom Conditions

exd42062 — Sat, 12 Oct 2013 12:17:28 GMT

I just tried your suggestion and after installing the app it gives me 4 errors while loading saying it cannot find search module and it has been loading for 15 minutes with no apparent progress. This won't work for me.

Re: Custom Conditions

exd42062 — Sat, 12 Oct 2013 12:26:02 GMT

Nevermind....I failed to read the doc about the dependencies...silly me. I got it installed properly now. I will see if it does what I want. I really wanted to learn the regex stuff, but maybe this will get me by for now. Thanks.