Getting alerts on sudden increase in traffic.

[vantoryc]

I have a custom script that collects stats on a custom HW appliance every minute and forwards it to our splunk system.

And has following style data:

log_type="throughput_data", local_time="2020/09/09 19:01 CST", server_ip="10.221.20.172", host_name="host2", host_ip="10.131.221.37", version="13", model="M1000", serial_no="1234234", ssl_card="No", total_traffic="93700", app_traffic="17524", cpu="15", ssl="0", http="258",connections="1", sql="0", sql2="0"
log_type="throughput_data", local_time="2020/09/09 19:01 CST", server_ip="10.221.20.172", host_name="host5", host_ip="10.131.222.36", version="13", model="M2000", serial_no="12342342", ssl_card="No", total_traffic="0", app_traffic="0", cpu="3", ssl="0", http="0",connections="0", sql="0", sql2="0"

I have a 2 parter question:

1. How do I go about generating an alert when the app_traffic has a sudden spike or out of usual spike.
   EG: normally the app_traffic hovers around 500 and there was a sudden increase to 10000.
   
   Just having this will make my team happy, but I do not believe that is the proper solution we need
   
   
2. Is there a way I can go about and create a dataset/lookup for each models supported datasheet values and generate an alert when that models certain values go up.
   EG: Model M1000 can do total app_traffic of 10000 and have an alert be generated when it reaches 90% of that value; in this case 9000.
   1. Can this be split do alert if either app_traffic or total_traffic or CPU or SSL reach 90% of the set limit in the data set
      
      I believe this will help us scale and be better for future use cases and making a business use case for management.

[vantoryc]

Thanks, Now I have a strong case to get our 6.5.2 upgraded to latest version.

[thambisetty]

I think below conf video will help you. finding outliers.

https://conf.splunk.com/files/2019/recordings/FN1390.mp4

[isoutamo]

Hi
You should look this conf presentation. https://conf.splunk.com/files/2016/slides/time-after-time-comparing-time-ranges-in-splunk.pdf
It gives you to good basement for this challenge.
r. Ismo