Dashboards & Visualizations

Only Uncleared Alert in dashboard

lord_prom
New Member

Suppose,

Alert name= link_down

event time 1st occured= 5pm
cleared time 1st occured = 6pm

event time 2nd occurred =7pm
(Still persisting..)

Normally I have created a dashboard with this particular alert but in dashboard three alerts captured. I wanna see the only one which is still persisting network (occurred at 7pm) not cleared alert. How can I do it...!

0 Karma

woodcock
Esteemed Legend

Here is the specific answer to go with the general one that I posted earlier:

| makeresults 
| eval raw="{\"prival\":\"187\", \"version\":\"2\", \"hostname\":\"X\", \"fromhost\":\"X\", \"fromhost-ip\":\"X.X.X.X\", \"app-name\":\"1234\", \"procid\":\"-\", \"structured-data\":\"-\", \"timestamp\":\"2019-05-16T00:30:07.009656+00:00\", \"time-received\":\"2019-05-16T00:30:07.009656+00:00\", \"msgid\":\"-\", \"severity\":\"ERR\", \"severity-value\":\"3\", \"facility\":\"LOCAL7\", \"tag\":\"91011:\", \"programname\":\"91011\", \"inputname\":\"imudp\", \"msg\":\" %LINK-1-UPDOWN : et -1/1/1, changed state to down \"}:::{\"prival\":\"187\", \"version\":\"2\", \"hostname\":\"X\", \"fromhost\":\"X\", \"fromhost-ip\":\"X.X.X.X\", \"app-name\":\"5678\", \"procid\":\"-\", \"structured-data\":\"-\", \"timestamp\":\"2019-05-16T00:30:04.994845+00:00\", \"time-received\":\"2019-05-16T00:30:04.994845+00:00\", \"msgid\":\"-\", \"severity\":\"ERR\", \"severity-value\":\"3\", \"facility\":\"LOCAL7\", \"tag\":\"91011:\", \"programname\":\"91011\", \"inputname\":\"imudp\", \"msg\":\" %LINK-1-UPDOWN : et -1/1/1, changed state to up \"}:::{\"prival\":\"187\", \"version\":\"2\", \"hostname\":\"X\", \"fromhost\":\"X\", \"fromhost-ip\":\"X.X.X.X\", \"app-name\":\"91011\", \"procid\":\"-\", \"structured-data\":\"-\", \"timestamp\":\"2019-05-16T00:30:03.010408+00:00\", \"time-received\":\"2019-05-16T00:30:03.010408+00:00\", \"msgid\":\"-\", \"severity\":\"ERR\", \"severity-value\":\"3\", \"facility\":\"LOCAL7\", \"tag\":\"91011:\", \"programname\":\"91011\", \"inputname\":\"imudp\", \"msg\":\" %LINK-1-UPDOWN : et -1/1/1, changed state to down \"}" 
| makemv delim=":::" raw 
| mvexpand raw 
| rename raw AS _raw 
| spath 
| eval _time = strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%6N%z") 
| sort 0 - _time 

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| streamstats count(eval(match(msg, "changed state to up"))) AS sessionID BY hostname fromhost "fromhost-ip"
| eventstats count BY sessionID hostname fromhost "fromhost-ip"
| where count==1
0 Karma

woodcock
Esteemed Legend

Very generally, like this:

index=<You should always specify an index> AND sourcetype=<And sourcetype too> AND (<alert string here> OR <clear string here>)
| streamstats count(eval(searchmatch(<clear string here>))) AS sessionID BY host <and maybe other fields here>
| stats count BY sessionID host <and maybe other fields here>
| where count==1
0 Karma

Sukisen1981
Champion

hi @lord_prom
you are just selecting your alerts based on the alert name. Your event examples are a bit generic, but what you need to do is:
1- Capture all event lines based on the alert name, on your example above this will give you 3 lines or more, if still persisting
2- define a new field , say something called status, which looks at each event line and checks if the word 'cleared' is present. if event line has cleared then status = cleared else status=occured
3-table the above with 3 columns, alert name,status,_time
4-do an eventstats over the _time...something like |eventstats max(_time) as maxtime by _time
5-check the status field where _time=maxtime...if the field has value cleared , alert has been cleared take no action if status=occured, then it is persisting and you have the latest (maxtime) of the alert when the alert last had occured, this is your last persistent time.

If you can share your sample logs with more exact data, we can give a more exact solution 🙂 but do consider the above approach .

0 Karma

lord_prom
New Member

Hi Sukisen,

Thanks for your feedback : Here are some logs - (Consider/ Source as prome_tomtom)

{"prival":"187", "version":"2", "hostname":"X", "fromhost":"X", "fromhost-ip":"X.X.X.X", "app-name":"1234", "procid":"-", "structured-data":"-", "timestamp":"2019-05-16T00:30:07.009656+00:00", "time-received":"2019-05-16T00:30:07.009656+00:00", "msgid":"-", "severity":"ERR", "severity-value":"3", "facility":"LOCAL7", "tag":"91011:", "programname":"91011", "inputname":"imudp", "msg":" %LINK-1-UPDOWN : et -1/1/1, changed state to down "}

{"prival":"187", "version":"2", "hostname":"X", "fromhost":"X", "fromhost-ip":"X.X.X.X", "app-name":"5678", "procid":"-", "structured-data":"-", "timestamp":"2019-05-16T00:30:04.994845+00:00", "time-received":"2019-05-16T00:30:04.994845+00:00", "msgid":"-", "severity":"ERR", "severity-value":"3", "facility":"LOCAL7", "tag":"91011:", "programname":"91011", "inputname":"imudp", "msg":" %LINK-1-UPDOWN : et -1/1/1, changed state to up "}

{"prival":"187", "version":"2", "hostname":"X", "fromhost":"X", "fromhost-ip":"X.X.X.X", "app-name":"91011", "procid":"-", "structured-data":"-", "timestamp":"2019-05-16T00:30:03.010408+00:00", "time-received":"2019-05-16T00:30:03.010408+00:00", "msgid":"-", "severity":"ERR", "severity-value":"3", "facility":"LOCAL7", "tag":"91011:", "programname":"91011", "inputname":"imudp", "msg":" %LINK-1-UPDOWN : et -1/1/1, changed state to down "}

Wonder if you would share your feedback

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...