Dashboards & Visualizations

how to define stop words in splunk

Explorer

Hi, I am doing some text mining research by splunk. Though the popular words can be detected, there are so much stop words, such as the, a, you, I, which means mining is nonsense for me. Does anybody know how to handle stop words in splunk?

Tags (2)
0 Karma
1 Solution

Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

View solution in original post

New Member

My case example:

eval text=lower(NOTES)|
eval text=urldecode(text)|
makemv text |
mvexpand text |top limit=50000 text|
search NOT ( [ |inputlookup StopWords_inc.csv|rename Word as text ] )|
head 100

PD: The NOTES field is used in my work to record Ticket data (service center)

0 Karma

SplunkTrust
SplunkTrust

If there's a question hiding in there please post it as a separate question.

0 Karma

Contributor

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

View solution in original post

Explorer

OK, Thank u. Then, I have another question how to search some word from both field Word in word.csv and field text.
I tried follow search command, but it turned out wrong

search text= [|inputlookup word.csv| fields word]
search text AND[|inputlookup word.csv| fields word]
0 Karma

New Member

Hi,
makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

is not working for me. I am getting no results. pls help.

0 Karma