topic Re: How to remove some words from multivalued field for display in Splunk Search

How to remove some words from multivalued field for display

manjushan — Mon, 20 Aug 2012 23:29:45 GMT

I have a field with search terms in the log that I want to break into individual words. But I do not want stop words (like A, AN, THE) etc to show up in the result.

eg: my log file has

searchTerm=lost+cat
searchTerm=lost+a+cat

searchTerm=lost+the_dog

And I want to get a list of all the words ever used in the searchTerm field.
lost
cat
dog

But I do not want "a" , "an", "the" to appear in the list.

I use makemv delim=" "

searchTerm | eval n = urldecode(searchTerm) | makemv delim=" " setsv=true n| top limit=10000 n

But I get all the words including "a", "the" etc. How do I get to NOT display these stop words.

Re: How to remove some words from multivalued field for display

lguinn2 — Tue, 21 Aug 2012 04:38:42 GMT

You could add in a where command before | top limit=10000 n

Like this

| where lower(n)!="a" AND lower(n)!="an" AND lower(n)!="the" | top limit=10000 n

If there are many words to eliminate, then you might want to use lookups instead.

Re: How to remove some words from multivalued field for display

manjushan — Tue, 21 Aug 2012 18:13:48 GMT

Thank You , It worked !