Anomaly detection search queries

[Dalton2]

Hi,

I'm trying to put together some search queries for some common anomaly detection. I've been trying to find ones for these issues and I seem to come up with nothing. 

Some common ones though would be:

• Drastic change in events per second
  
  
• Drastic change in size of events
  
  
• Blocked outputs
  
  
• Change of health of inputs?
  
  
• Dropped events
  
  
• CPU percentage use that is REALLY high (percentage?)

[ITWhisperer]

This is a vague question with a multitude of possible answers, however, there are a couple of techniques ranging from the simplistic to more complex.

For a simplistic approach, you could determine the (historic) average of each of your metrics and compare your current values against that average. If you also determine the standard deviation of your metrics, your comparison can be based on number of standard deviations away from the mean that your current values are. You would then set a threshold for how far from the mean would be deemed an anomaly.

More sophisticated ways of doing this is to use the Machine Learning ToolKit (MLTK) - this involves fitting your (historic) data to a statistical model, and then applying that model to your current data to find anomalies. The MLTK can fit your data to a number of different distribution models either specifically if you know the type of distribution your data is expected to follow, or let the MLTK find the most appropriate.

[gcusello · Accepted solution]

HI @Dalton2,

let me understand: do you want to have in the same table the values of your ix use cases that you already have, or do you want a solution to each of the six use cases?

in the first case,

• you should create six searches that have as output two columns: UseCase (containing the use cases you have) and value (containing the fould values);
• than store the results of these six searches in a summary index using the collect command;
• then you can search in the summary index and use the two stored columns to display results in a table.

In the secod case, you want six different use cases that depend on many factors like kind of data, fields, etc... and it's too long for one answer and I hint to divide it in more questions.

Ciao.

Giuseppe

[Dalton2]

Hi,

I'm saying for these issues you've answered some of it. What I was reaching out to the community for was search queries for each of these issues. I'm trying to use different types of search queries and can't seem to get something to stick for each of those issues. I'm trying to make a table for each one of those issues but if columns for those issues you think would be better then I'll experiment with that idea as well. I just can't seem to get any to show up. I'm using them for note purposes. I'm just needing assistance from someone being able to show me how to get search queries for each.