Splunk Search

Custom Conditions

exd42062
Path Finder

I am new to splunk and want to create an alert for switch ports that flap. So, I need a way to alert if a port on a given switch goes up and down more than a certain number of times per minute. How can I setup the search/alert for that? Below is an example of the search results.

Oct  9 12:01:18 hos-a-3550-1.rockefeller.internal 2635634: Oct  9 12:01:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/3, changed state to down
1 Solution

exd42062
Path Finder

mysearch | rex "Interface (?[^, ]*), changed state to (?\w+)" | table interface state

I saw your other post looking to help me with this. Thanks. YannK seems to know this stuff pretty well. This should be enough to get me started. Not sure why the IFX did not work well, but I can figure that out later.

View solution in original post

0 Karma

exd42062
Path Finder

mysearch | rex "Interface (?[^, ]*), changed state to (?\w+)" | table interface state

I saw your other post looking to help me with this. Thanks. YannK seems to know this stuff pretty well. This should be enough to get me started. Not sure why the IFX did not work well, but I can figure that out later.

0 Karma

mikaelbje
Motivator

Why don't you try the Cisco IOS app out? All the extractions are there. You just need to set up the alert.

0 Karma

exd42062
Path Finder

Nevermind....I failed to read the doc about the dependencies...silly me. I got it installed properly now. I will see if it does what I want. I really wanted to learn the regex stuff, but maybe this will get me by for now. Thanks.

0 Karma

exd42062
Path Finder

I just tried your suggestion and after installing the app it gives me 4 errors while loading saying it cannot find search module and it has been loading for 15 minutes with no apparent progress. This won't work for me.

0 Karma

twinspop
Influencer

If you're going to search over a time range and want to know if you were flapping during any 1 minute in the time range:

<your search> | bucket span=1m _time | stats count by interface host _time | where count>1000

If you just want to check the last minute:

<your search> earliest=-1m@m latest=@m | stats count by interface host | where count>1000
0 Karma

lukejadamec
Super Champion

Try this and see if it gives you the results you want.

 index=yourindex sourcetype=yoursourcetype interface="*" up-down="*" |stats count by interface,up-down | where count>1000 |table interface,up-down,count

Change the 1000 to what ever value you want.

From the time picker on the right, select real time 1 minute window.
From the Create dropdown on the right, select Alert.

If you want to extract the fields in the search then you can use rex:

index=yourindex sourcetype=yoursourcetype | rex "interface (?<Interface>[^, ]*), changed state to (?<state>\w+)" |stats count by Interface,state | where count>1000| table Interface state

lukejadamec
Super Champion

Perhaps you need an inline field extraction.

0 Karma

exd42062
Path Finder

I am typing the field name right...it does not show up. The time frame is plenty, since i'm seeing these literally every second. I tested the extraction while creating it and it worked so I'm confused why it won't work with a search. The interface extraction works fine.

0 Karma

lukejadamec
Super Champion

The up-down field should be included in the field list to the right. The field name will be case sensitive.
Also, for testing you should search a large enough timespan to ensure you are getting some events with up-down values.

0 Karma

exd42062
Path Finder

While I wating, I tried the * and that worked for the "interface" field I created, but not for the "up-down" field. Not sure the up-down field is working right.

0 Karma

lukejadamec
Super Champion

I'm heading offline for about 8 hours. I know this works because I tested on windows EventCodes,ComputerName log entries per minute. Good luck.

0 Karma

lukejadamec
Super Champion

The way this search works is it works with your log event triggers. For any particular interface you will only get this event when the state changes from up to down or down to up, so for any particular interface over the course of a minute you get >x number of state changes it will show up in the table. You decide what x should be.

Set the number to 0 for testing.

0 Karma

lukejadamec
Super Champion

Sorry, the * were stripped out. I'll fix it.

0 Karma

exd42062
Path Finder

Am i supposed to put something in the quotes? I get no results.

I could have any number of different interfaces go up and down and I want to see the ones that go up and down a lot. I don't care about the ones that just up and down once or twice.

0 Karma

exd42062
Path Finder

In Field Extractions I've got "interface" and 'up-down".

0 Karma

lukejadamec
Super Champion

We can help with that. I'm not the best with this sort of search, but a lot of folks are.
What did you name the fields?

0 Karma

exd42062
Path Finder

I created a field extraction for FastEthernet0/3 and when I test it gives me the right list of values. I also created an even type for the Up/Down events. My problem is I don't know what to do next. Do I need to extract the Etehrnet field and the Up/Down together in one field or separate fields. And then I'm not sure what stats to run. I've read about count and distinct count ,but I'm not sure how to formulate the query from there.

0 Karma

lukejadamec
Super Champion

That should be pretty easy then. You'll need to use rex to extract the FastEthernet0/3 and the 'changed to down/up' information as fields in the search, then you can run statistics on them, and alert when the number for an interface is higher than you'd like.

Try to create the extractions with Field Extractor (down arrow key next to the event). It is easy to use and works most of the time.

0 Karma

exd42062
Path Finder

single field

0 Karma

lukejadamec
Super Champion

Are there any fields in this event, or is this string the contents of a single field?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...