anomaly detection for multiple fields

[gbenor]

Hi,

I would appreciate your help in implementing the following alert with Splunk and the machine-learning toolkit.

Let's start with a simple example. Suppose I have one host in my system which sends one of two predefined messages. Then, the event should consist of two fields: [_time, message]. I can use the timechart command to generate two new numerical timeseries:

1. count of total events.
2. count of each predefined message.

Finally, I can use the machine learning toolkit to detect outliers and anomalies.

Now, I would like to describe my real situation: I have an unknown number of hosts; each host may send any kind of message. A typical event looks like: [_time, host, message].

I would like to implement an outlier alert for each possible host, possible message, and for the total number of messages per host. I prefer to have a single alert for all combinations of host and message_type. In addition, I would like to have a visualization of the timeseries of each combination.

Unfortunately, I don't have a clue how to implement this task in SPL. 

A python solution may look like the following:

1. find unique hosts.
2. find unique messages.
3. For host in hosts:
1. For msg in messages:
1. Do anomaly detection (host, msg)
2. Do anomaly detection (host, msg_count)