Here is my requirement
I have file with column 'Description'. I need to get the most common pattern of the words.Example
Repetitive Pattern Count Percentage Examples
Job 80 15% Job Related with Ticket number
Access 130 20% Access issues
Any Job or Jobs should categorize as Job.
I have installed Machine Learning Tool Kit and tried to apply TFIDF and Kmeans. I am unable to proceed as i am new to splunk.
Can any one help me how to do clustering using Kmeans with data as mentioned above and get required output.
You can use the
kmeans command for this:
Or you can have even more control in the
Machine Learning ToolKit (MLTK) to build a model. Once that is done, you can inspect the KMeans model you built with fit using the summary command:
| summary <your_model_name>
When assigning new points to the appropriate cluster, you can simply apply your model like this:
<new_points> | apply <your_model_name>
I have tried below search command to exclude stop words
index=sample| makemv Summary | mvexpand Summary|fields Summary| search Summary NOT [|inputlookup words.csv|rename word as summary1]|top summary1
No results are fetched. Please help where i am doing mistake
If you don't have a list of keywords, you can try the
But it sounds like you have a limited set so you can do something like this:
Your Search Here | eval cluster_keyword = case( match(_raw, "(?i)job"), "job", match(_raw, "(?i)access"), "access", match(_raw, "(?i)ticket"), "ticket", true() "other") | stats first(_raw) last(_raw) count BY cluster_keyword | eventstats sum(count)AS total | eval pct = 100 * count / total
Thank you for the quick reply.
Firstly, want to remove the stop words and categorize the similar words into one category. Next should be, most recurrent words should display with count.
How can i implement this logic in Splunk. I need to use Kmeans algorithm
For string matching you could check this post:
Intelligent text pattern matching might be a little hard to implement. I will investigate further.
Hope it helps!!!