how to define stop words in splunk

[mhqssyh]

Hi, I am doing some text mining research by splunk. Though the popular words can be detected, there are so much stop words, such as the, a, you, I, which means mining is nonsense for me. Does anybody know how to handle stop words in splunk?

[yserrano]

My case example:

eval text=lower(NOTES)|
eval text=urldecode(text)|
makemv text |
mvexpand text |top limit=50000 text|
search NOT ( [ |inputlookup StopWords_inc.csv|rename Word as text ] )|
head 100

PD: The NOTES field is used in my work to record Ticket data (service center)

[martin_mueller]

If there's a question hiding in there please post it as a separate question.

[jimodonald · Accepted solution]

Chapter 12 of "Big Data Analytics Using Splunk" covers the details of what you are looking to do. In the book, the author is analyzing a Twitter feed.

Some quick outtakes:
Most popular words (page 220)

* | head 200 | makemv text | mvexpand text | top text

and filtering out the stop words (page 221)

lang=en | makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

Hope that points you in the right direction.

[mhqssyh]

OK, Thank u. Then, I have another question how to search some word from both field Word in word.csv and field text.
I tried follow search command, but it turned out wrong

search text= [|inputlookup word.csv| fields word]
search text AND[|inputlookup word.csv| fields word]

[lavanya_gurrapu]

Hi,
makemv text | mvexpand text | search text NOT [ |inputlookup StopWords.csv | rename Word as text ] | top text

is not working for me. I am getting no results. pls help.