topic Re: How to find most common words used by cluster command in All Apps and Add-ons

How to find most common words used by cluster command

edoardo_vicendo — Mon, 11 Feb 2019 11:57:29 GMT

Hi All,

We are trying to cluster a Description field with cluster command in this way:

| cluster t=0.5 labelonly=t showcount=t field=Description match=termset
| table cluster_label cluster_count Description

Do you know if is it feasible to know which are the most common words or string, on each cluster, the algoithm has used on the Dataset to generate the clusterization?
I would like to add one column on the right to my output table containing the patterns that have generate this cluster (that are linked to Description field).

cluster_count | cluster_label | Description             | words
         120  |            1  | Bla bla bla ciao ciao   | bla, ciao
          80  |            2  | Day after day is better | Day

Thanks a lot,
Edoardo

Re: How to find most common words used by cluster command

grana_splunk — Mon, 11 Feb 2019 18:19:28 GMT

Hi,

Did you tried TFIDF algorithm?
https://docs.splunk.com/Documentation/MLApp/4.1.0/User/Algorithms#TFIDF

Additionally, if you are interested in Text Analytics usecase, I would recommend to look into NLP Text Analytics app which is using MLTK Algorithms: https://splunkbase.splunk.com/app/4066/#/details

Let me know if it helps.

Thanks,

Re: How to find most common words used by cluster command

mschaaf — Wed, 07 Aug 2019 18:47:40 GMT

After your cluster command, pipe to the findkeywords command. It will add fields "includeKeywords" and "excludeKeywords," among others, which indicate the terms used for each cluster_label.

Re: How to find most common words used by cluster command

sijingwu — Tue, 10 Sep 2019 19:37:25 GMT

hi, could you please explain how the findkeywords command works? I couldn't find it anywhere on the Splunk documents.

Re: How to find most common words used by cluster command

skoelpin — Tue, 10 Sep 2019 20:21:34 GMT

Yep! x2 for the NLP app. It gives you the cleantext command

Re: How to find most common words used by cluster command

mschaaf — Wed, 11 Sep 2019 13:37:09 GMT

https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Findkeywords