Dashboards & Visualizations

how to define stop words in splunk

mhqssyh
Explorer

Hi, I am doing some text mining research by splunk. Though the popular words can be detected, there are so much stop words, such as the, a, you, I, which means mining is nonsense for me. Does anybody know how to handle stop words in splunk?

Tags (2)
0 Karma
1 Solution

jimodonald
Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

View solution in original post

yserrano
New Member

My case example:

eval text=lower(NOTES)|
eval text=urldecode(text)|
makemv text |
mvexpand text |top limit=50000 text|
search NOT ( [ |inputlookup StopWords_inc.csv|rename Word as text ] )|
head 100

PD: The NOTES field is used in my work to record Ticket data (service center)

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If there's a question hiding in there please post it as a separate question.

0 Karma

jimodonald
Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

mhqssyh
Explorer

OK, Thank u. Then, I have another question how to search some word from both field Word in word.csv and field text.
I tried follow search command, but it turned out wrong

search text= [|inputlookup word.csv| fields word]
search text AND[|inputlookup word.csv| fields word]
0 Karma

lavanya_gurrapu
New Member

Hi,
makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

is not working for me. I am getting no results. pls help.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...