SPLUNK Search if word form file appears in logs

[swdowiarz]

Hi

I have the following issue. I'm using SPLUNK for real-time monitoring of chat bot. I have as well file with banned words, let say that's a csv file. I want to do search to check if any of those words appears in logs.
What is the best way to do it ? how it would affect on a performance ?

I would be rally grateful for any piece of information.

[DavidHourani]

Hello swdowiarz,

I recommend following these steps:

1- start by creating an index for your data.
2- Get your data into this index.
3- Make a list of bad words that you want to check for and put it in a csv lookup file, lets call it badWords.csv.
4- Create a scheduled search (report), that checks if your data has any of the words in the lookup file.

Your search would look like this if you want to look anywhere in your raw data for the bad words:

index=yourIndexName [|inputlookup badWords.csv |fields badWords |rename badWords AS _raw|rex field=_raw mode=sed "s/(.+)/*\1*/" |return 10000 _raw]

If you are looking for the words in a specific field. lets call it fieldGoodWordsOnly then your search should be:

 index=yourIndexName [|inputlookup badWords.csv | fields badWords| rename badWords as fieldGoodWordsOnly |return 10000 fieldGoodWordsOnly ]

Let me know if that helps.

Best regards,
David

[gcusello · Accepted solution]

Hi swdowiarz,
real time searches are always more expensive of normal searches!
If compliant with your requirements, I suggest to schedule an alert (every e.g. 5 minutes) checking the presence of one of the words of your csv using a search like this (csv file is called "my_patterns.csv" and it has only a field called "word"):

| your_search [ | inputlookup my_patterns.csv | rename word AS query | fields query ]
| ....

Otherwise, if your requirements need a real-time search, use it but monit your system for performaces (especially check index queues).

Bye.
Giuseppe

[swdowiarz]

I've added new lookup from my csv file which contains only words like:

sebastian
firstword
secondword
thirdword
fourthword

now I want to do a search to check if any of my events contains any of those words which are in lookup ?

[gcusello]

Hi swdowiarz,,
use the search I suggested in my previous message, the most important thing is the rename command in the subsearch: the field name in your lookup must be renamed in "query", in other words try:

| your_search [ | inputlookup my_patterns.csv | rename word AS query | fields query ]
| ...

Bye.
Giuseppe

[swdowiarz]

Awesome job ! That is what I want ! Thank you so much for help ! All the best for you !

[Elsurion]

Since you search your logfile realtime, the events will be processed realtime. The performance impact is depending on the eventflow. When only a few events per minute, there's no impact at all. When you do a few thousands a second, then you probably will have a performance issue.

You get the whole message as a field or just a simple text?
At the moment i have no idea to do this inline in the SPL, what you could do is with an custom script within your search.

http://docs.splunk.com/Documentation/Splunk/7.0.1/SearchReference/Script

[risgupta]

Hi ,

Let's say it is "abc.csv" file for your data. Then you can upload your file to a lookups, after navigating it to
Settings--> Lookups--> Add new
then browse your file and add to splunk and name it like "abc.csv" once it is done, then you can go to splunk search and type like:

|inputlookup abc.csv |search "Your keyword in your file"


Let me know how it goes.

[swdowiarz]

more less it is something what I want to achieve, I've added new lookup which is my csv file with banned words, but how can I now see result of events which contains any of those words ?

[swdowiarz]

my csv file contain only words like:

sebastian
firstword
secondword
thirdword
fourthword