anomaly detection for multiple fields

New Member


I would appreciate your help in implementing the following alert with Splunk and the machine-learning toolkit.

Let's start with a simple example. Suppose I have one host in my system which sends one of two predefined messages. Then, the event should consist of two fields: [_time, message]. I can use the timechart command to generate two new numerical timeseries:

  1. count of total events.
  2. count of each predefined message.

Finally, I can use the machine learning toolkit to detect outliers and anomalies.


Now, I would like to describe my real situation: I have an unknown number of hosts; each host may send any kind of message. A typical event looks like: [_time, host, message].

I would like to implement an outlier alert for each possible host, possible message, and for the total number of messages per host. I prefer to have a single alert for all combinations of host and message_type. In addition, I would like to have a visualization of the timeseries of each combination.

Unfortunately, I don't have a clue how to implement this task in SPL. 

A python solution may look like the following:

  1. find unique hosts.
  2. find unique messages.
  3. For host in hosts:
    1. For msg in messages:
      1. Do anomaly detection (host, msg)
    2. Do anomaly detection (host, msg_count)









Labels (1)
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.