topic Re: How can I calculate the term frequency for all the words in a field's values? in Splunk Search

How can I calculate the term frequency for all the words in a field's values?

mhqssyh — Mon, 29 Dec 2014 10:29:53 GMT

I am trying to calculate some term frequency on the field. The field is defined as follow.
rex field=_raw "Notes : (?.*)"
And, the field is generated correctly, but it hasn't any format, such as:

Notes :

Notes : Troubleshooting, I am simply reinstalling.
Notes : program would not start. I am reinstalling.
Notes : Made MacBook too slow
Notes : computer to slow when using the program! Need to install it into another!

There are thousands line of information, and I want to know the term frequency of all the words in the field of notes. I'd like to know whether there is a command to do this, or how can I achieve this in splunk.

Any ideas?
Thanks, Yi

Re: How can I calculate the term frequency for all the words in a field's values?

jimodonald — Mon, 29 Dec 2014 18:18:10 GMT

I think the REX from that post should get you going in the right direction. I've pasted the REX below. Please see the original for more details.

source=*mybook* | sort -_time | rex mode=sed "s/(\.|,|;|=|\"|'|\(|\)|\[|\]| -|!|^-)/ /g" | eval word=_raw | makemv delim=" " word | mvexpand word | eval word=lower(word) | eval position=1 | streamstats sum(position) AS position | table position word | stats count min(position) max(position) by word

Re: How can I calculate the term frequency for all the words in a field's values?

mhqssyh — Tue, 30 Dec 2014 01:54:18 GMT

Thanks, jimodonald! I tried the REX. It works. But now I have another question that can I cluster some similar words to one class, such as fast, quick, rapid, swift.

Re: How can I calculate the term frequency for all the words in a field's values?

jimodonald — Tue, 30 Dec 2014 16:27:59 GMT

Splunk is not going to know what words are synonyms. It could likely be done with a case statement or a lookup table. Either way the synonyms would need to be identified and linked back to a common word.

Re: How can I calculate the term frequency for all the words in a field's values?

jzapantis — Mon, 03 Apr 2017 21:01:39 GMT

you have to use a lexicon. Look up the nodejs library for Word Net. Upload that library. Then build a new app in splunk. Once that is done, create a .js file that calls the word net library, then define a search manager in the .js file that returns your splunk search. Loop through all the words, and pass each one to the word net library to built a temporary sysnonym dictionary. You can optionally save this dictionary as a KV store and continually update.

I know I didnt give details, but thats because it is a highly involved solution. But it is possible. Start poking around with Word Net and the capabilities.

Keep in mind, that all custom Splunk apps are basically Node.js apps - at least that is my current understanding. Community, let me know if I am wrong!

Re: How can I calculate the term frequency for all the words in a field's values?

jzapantis — Tue, 04 Apr 2017 15:54:48 GMT

I used the following syntax to count the frequency of terms in my field:

              | rename COMMENTS_4 AS text
              | rex mode=sed field=text "s/[,|.|!]/ /"
              | makemv text
              | mvexpand text
              | eval wordCount = mvcount(text)
              | stats sum(wordCount) as "Word Map Text Analysis" by text

the line: | rename COMMENTS_4 AS text
just names my field variable to "text". So assuming you rename your field variable with text, you can count the terms using MV* commands