With help from the Splunk Machine Learning Toolkit, I've constructed a query that detects numeric outliers; in this case the sum of outbound bytes from a server in 10 minute chunks:
index="proxies" earliest=-7d@d (c_ip="1.2.3.4")
| timechart span=10m sum(bytes_out) as bytesout by c_ip
| streamstats window=200 current=true median("1.2.3.4") as median
| eval absDev=(abs('1.2.3.4'-median))
| streamstats window=200 current=true median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(20)), upperBound=(median+medianAbsDev*exact(20))
| eval isOutlier=if('1.2.3.4' < lowerBound OR '1.2.3.4' > upperBound, 1, 0)
| where isOutlier=1
Now, what I would like to do is iterate over a group of servers like this:
index="proxies" earliest=-7d@d [|inputlookup lu_inventory where function="web_server" | table ip | rename ip as c_ip | format]
| foreach c_ip [| timechart span=10m sum(bytes_out) as bytesout by '<<FIELD>>'
| streamstats window=200 current=true median('<<FIELD>>') as median
| eval absDev=(abs('<<FIELD>>'-median))
| streamstats window=200 current=true median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(20)), upperBound=(median+medianAbsDev*exact(20))
| eval isOutlier=if('<<FIELD>>' < lowerBound OR '<<FIELD>>' > upperBound, 1, 0)
| where isOutlier=1]
But, the problem is the foreach command cannot contain non-streaming commands.
So, is there a way to programmatically iterate over a list of IPs and find the outliers?
... View more