I have a case where a "Message" field contains sentences of strings, which indicated different kind of system errors.
We want to use the machine learning toolkit to automatically clusters those errors into several categories.
Since we are dealing with sentences, we first decided to use TFIDF to vectorize the strings, and then use the DBSCAN to do the clustering. Here is the search:
index="mail" sourcetype="P1_tickets"
| rex field=_raw "Message\s+:(?<Message>(.*\n)+?(?=Extra Message|Control|Log|Repeats|Via Host))"
| fit TFIDF Message into message_model
| fit DBSCAN Message_tf*
The result is promising. We are seeing similar system errors being grouped together into the same cluster. However, the clusters are named by default 0.0, 1.0, 2.0, and etc. We want to actually use the keywords from the sentences to name the clusters, which can actually give the user some idea of what the error is. Is there anyway to achieve this in Splunk?
