Hi,
I am using the Network Toolkit to ping site routers on our MPLS network.
We currently have 3 Runs (pings) every 1m to the routers IP address.
We then have a real-time alert as follows:
sourcetype="ping_input" packet_loss=100
| stats max(dest) as Host max(packet_loss) as "Packet Loss"
We are getting lots of results back for 100% packet loss, however if we run a continual ping from the Splunk Server (centos) we dont get any packet loss?
Looking for some assistance with how this should be setup?
thanks
David
The odd packet can always get a bit lost along the way, so you may want to use stats avg(packet_loss)
and set a threshold |where "Packet Loss">75
to avoid some of the false positives.
You also probably want to avoid real time scheduled searches (see: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html )
instead try using a historic search earliest=-2m@m latest=-1m@m
sourcetype="ping_input" earliest=-2m@m latest=-1m@m
| stats max(dest) as Host avg(packet_loss) as "Packet Loss"
|where "Packet Loss">75
The odd packet can always get a bit lost along the way, so you may want to use stats avg(packet_loss)
and set a threshold |where "Packet Loss">75
to avoid some of the false positives.
You also probably want to avoid real time scheduled searches (see: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html )
instead try using a historic search earliest=-2m@m latest=-1m@m
sourcetype="ping_input" earliest=-2m@m latest=-1m@m
| stats max(dest) as Host avg(packet_loss) as "Packet Loss"
|where "Packet Loss">75
Thanks Nick.
I have updated the Search and will monitor it.
Appreciate the link too.
Hi Nick,
Apologies for the delayed response.
That search brings up the desired results however the Alert I configured from it doesn't seem to trigger?
Have set the Alert with the same time frame as the search.
Any ideas?
Alert Type = Scheduled
Run Cron = earliest 2m latest 1m
Number of results > 0
thanks David
Are you using -2m
or just 2m
?
You need the -
yes I have -2m and -1m
If you run a historical search in Splunk over the last hour or so - do you still see results for packet loss?
If so, can you post some of the events?
Hi Nick,
Thanks for responding.
I searched for the last hour and got one result which was True as the host was restarted.
I expanded the search for Today and got some results which was a False Positive, but no others?
Looking through the amount of Alerts I received today I had 19.
Here are some events:
25/03/2019
13:30:04.000
sent=1 received=0 packet_loss=100 min_ping=NA avg_ping=NA max_ping=NA jitter=NA return_code=1 dest=router01.megt.com.au
host = splunk source = ping sourcetype = ping_input
25/03/2019
13:01:03.000
sent=1 received=0 packet_loss=100 min_ping=NA avg_ping=NA max_ping=NA jitter=NA return_code=1 dest=router02.megt.com.au
host = splunk source = ping sourcetype = ping_input
Thanks