topic Re: How to detect and alert abnormal spikes of web api requests in Alerting

How to detect and alert abnormal spikes of web api requests

bestSplunker — Tue, 02 Jul 2019 08:05:57 GMT

hi. everyone .

My website has some API interfaces. Sometimes malicious attacks will request these api continuously. It is clear on the time chart that the peak has been reached. How do I detect and alert?

for example:

I have a search like this now, I can see the number of requests per hour for these URIs.

index = web   sourcetype=nginx_access   uri=/api/getuserInfo   OR uri=/api/featchData OR uri=/login  OR uri=/home
|timechart span=1h count by uri

under normal conditions the request per hour of /api/getuserInfo is about 1000~5000 times, if a certain time period encounters a malicious attack, the interface requests 50,000 times. I think this is an anomaly. How should I use a smarter method to detect abnormal peaks and issue alarms?

I think of a stupid way, i can write the number of api interface requests per hour to csv or kvstore, and then use today's and yesterday's comparisons to see the magnitude of the rise. If the rise is too high, I think this is abnormal peaks

But I think there are more efficient methods, such as machine learning? Can someone help me and share a use case with me, thank you

Note: I have a lot of API interfaces, about 20, I want to monitor the abnormal peak of each API interface,

Re: How to detect and alert abnormal spikes of web api requests

woodcock — Tue, 02 Jul 2019 14:23:59 GMT

Check this out:
https://answers.splunk.com/answers/511894/how-to-use-the-timewrap-command-and-set-an-alert-f.html

Re: How to detect and alert abnormal spikes of web api requests

niketn — Wed, 03 Jul 2019 01:58:13 GMT

@bestSplunker you can refer to Detect Numerical Outlier examples in Splunk Machine Learning Toolkit to check out methods to find outliers and choose the best one as per your data needs. Refer to recent answer: https://answers.splunk.com/answers/754085/calculating-median-of-count-over-time.html

Even if you do not install MLTK in your production system you can use the SPL directly (generated by MLTK in non-prod system) along with Splunk's built in visualization to depict the outliers. Refer to an older answer of mine: https://answers.splunk.com/answers/747177/how-to-add-a-reference-line-to-an-outlier-chart-cr.html