Hi, We are sending a reduced size logs to out splunk to do some smarts. We realized for the past year or so one of our alerts is not working at all. Between that year we have upgraded splunk from 6.5.2 to latest 8.2.1 and also migrated it from the entire VM it sits on.   index=clean_security_events earliest=-1h | stats count as Events by SG mx_ip | join SG [search index=clean_security_events earliest=-720h latest=-1h | bin span=1h _time | stats count by SG _time | streamstats mean(count) as Average, stdev(count) as Deviation, max(count) as Peak by SG | dedup SG sortby -_time | eval Average = round(Average) | eval Variance = Deviation / Average] | where Events > (Average + (Deviation * (Variance + 10))) AND Events > (Average * 20) AND Events > 20000 AND Events > Peak AND Average > 50 | lookup mx2cx mx_ip | table ServerGroup mx_ip cx Events Average   The general idea is we send reduced security events from our app and use the above to determine if a given SG (hence the stat count as Events) is generating sudden high events compared to the last 30 days. Upon trial and error if I narrow down to one mx_ip out of the 100s it works. I suspect that the subsearch is either generating too many events or the result are taking too long for the parent search and as a result we are getting empty tables. Any idea how to fix this? My understanding is I can increase the limits but it is not recommended.  I was thinking to use some ML toolkit to detect outlier and that way I can replace two alerts (one for sudden uptick and one for sudden downtick) ... View more

I have a custom script that collects stats on a custom HW appliance every minute and forwards it to our splunk system. And has following style data:     log_type="throughput_data", local_time="2020/09/09 19:01 CST", server_ip="10.221.20.172", host_name="host2", host_ip="10.131.221.37", version="13", model="M1000", serial_no="1234234", ssl_card="No", total_traffic="93700", app_traffic="17524", cpu="15", ssl="0", http="258",connections="1", sql="0", sql2="0" log_type="throughput_data", local_time="2020/09/09 19:01 CST", server_ip="10.221.20.172", host_name="host5", host_ip="10.131.222.36", version="13", model="M2000", serial_no="12342342", ssl_card="No", total_traffic="0", app_traffic="0", cpu="3", ssl="0", http="0",connections="0", sql="0", sql2="0"       I have a 2 parter question: How do I go about generating an alert when the app_traffic has a sudden spike or out of usual spike. EG: normally the app_traffic hovers around 500 and there was a sudden increase to 10000. Just having this will make my team happy, but I do not believe that is the proper solution we need Is there a way I can go about and create a dataset/lookup for each models supported datasheet values and generate an alert when that models certain values go up. EG: Model M1000 can do total app_traffic of 10000 and have an alert be generated when it reaches 90% of that value; in this case 9000. Can this be split do alert if either app_traffic or total_traffic or CPU or SSL reach 90% of the set limit in the data set I believe this will help us scale and be better for future use cases and making a business use case for management.       ... View more