Hi,
I would appreciate your help in implementing the following alert with Splunk and the machine-learning toolkit.
Let's start with a simple example. Suppose I have one host in my system which sends one of two predefined messages. Then, the event should consist of two fields: [_time, message]. I can use the timechart command to generate two new numerical timeseries:
- count of total events.
- count of each predefined message.
Finally, I can use the machine learning toolkit to detect outliers and anomalies.
Now, I would like to describe my real situation: I have an unknown number of hosts; each host may send any kind of message. A typical event looks like: [_time, host, message].
I would like to implement an outlier alert for each possible host, possible message, and for the total number of messages per host. I prefer to have a single alert for all combinations of host and message_type. In addition, I would like to have a visualization of the timeseries of each combination.
Unfortunately, I don't have a clue how to implement this task in SPL.
A python solution may look like the following:
- find unique hosts.
- find unique messages.
- For host in hosts:
- For msg in messages:
- Do anomaly detection (host, msg)
- Do anomaly detection (host, msg_count)
