Splunk Search

Using lookup to exclude a list of Service_File_Names

cburr2012
Path Finder

Hello,

After some time spent Googling/Splunking yesterday, I could not find a unique solution to my problem.

Goal: I have a list of services (splunkd.exe, splunkweb.exe, svchost.exe, etc) that I want to exclude from a search without having a query string that is 25 lines. I want to read the exclusion list from a CSV.

Query pseudocode: Search index to see if any services have been installed on any systems. EXCLUDE services from .

Good things: If I just have | inputlookup this_lookup | fields services, then I can see all of my values of that field in a table in splunk.

Bad things: If I say NOT | inputlookup this_lookup | fields services | It doesn't recognize the match between the values in the CSV and the service_file_names in the logs, returns ALL results.

Bottom line (& at the end of the day? ;)): I noticed that in the events, the Service_File_Name is a full path, i.e. C:\Win\temp\this.sys, sometimes with quotes, sometimes with %sys%. Is it possible that the formatting I have stored these service file names in is not appropriate? It works as a query to say NOT (this.sys OR mcafee.exe), so I used the same formatting for those service names in my CSV.

Looking for direction, thank you.

P.S. I've looked at almost all of the inputlookup questions (and read the documentation) and haven't found my solution.

Tags (2)
1 Solution

Ayn
Legend

I suspect something is wrongly formatted in your lookup, because the inputlookup approach you've used is what I was just about to suggest.

<any other search parameters> NOT [| inputlookup this_lookup | fields services]

To look at how Splunk sees what comes from the lookup, you could run the subsearch as its own search and add format to the end:

| inputlookup this_lookup | fields services | format

This is exactly what the subsearch will get substituted with, so if you take the search string that you get from format and it doesn't work correctly when you enter it directly into the search field, then there's your problem. Just note that you need to think about negating this with an initial NOT to really try the same thing as in your first search.

View solution in original post

Gilberto_Castil
Splunk Employee
Splunk Employee

You are on the right track. Perhaps a slightly different approach will help. Why not create a blacklist of services in your lookup table. That means your CSV is named "service_black_list.csv" an it has content like this:

service_name, exclude
splunkd.exe, true
splunkweb.exe, true
svchost,exe, true

When you make the association with the lookup, you should ensure that the default value is false.

props.conf  

[WinEventLog:Security]  
LOOKUP-service_black_list = service_black_list service_name AS service_name OUTPUTNEW

transforms.conf

[service_black_list]  
filename = service_black_list.csv  
min_matches = 1  
default_match = false  

The upshot benefit then is that your seach becomes simple, like this:

sourcetype="WinEventLog:Security" service_name="*" exclude="false"

This will list only those services that are _not listed in the CSV.

I hope this helps.

lguinn2
Legend

Yes, if the Service_File_Name is a full path, your lookup table should contain the full path, too. There is a way around this, but it is just much easier to set up the lookup table for an exact match.

For this solution, put a second field in your lookup table. I called it "Matched". For each service, I put a value of "yes". So the table looks like

Service_File_Name,Matched  
servicepath1,yes  
servicepath2,yes  
...  

In my example, I have called the lookup service_lookup. You should also set the Advanced Options on the lookup, so that the min and max matches = 1 and the default value is NoMatch.

yoursearchhere
| lookup service_lookup Service_File_Name OUTPUT Matched
| where Matched="NoMatch"

It would probably be more efficient to do the search as you originally imagined it, but this will work.This search will eliminate events with a Service_File_Name that appears in the lookup table.

Ayn
Legend

I suspect something is wrongly formatted in your lookup, because the inputlookup approach you've used is what I was just about to suggest.

<any other search parameters> NOT [| inputlookup this_lookup | fields services]

To look at how Splunk sees what comes from the lookup, you could run the subsearch as its own search and add format to the end:

| inputlookup this_lookup | fields services | format

This is exactly what the subsearch will get substituted with, so if you take the search string that you get from format and it doesn't work correctly when you enter it directly into the search field, then there's your problem. Just note that you need to think about negating this with an initial NOT to really try the same thing as in your first search.

View solution in original post

cburr2012
Path Finder

lguinn,

It is working. Simple, but, working. I am very new to Splunk and want to get into good habits.

What is the benefit to using output?

0 Karma

lguinn2
Legend

Yes, you can put a * in the CSV file - if you are using it this way. Glad to see that this option is working for you as well.

0 Karma

cburr2012
Path Finder

Ayn,

Thanks for the response.

I did as you advised and I noticed that it is searching for "services=this.exe" which is obviously not the same as Splunk's "Service_File_Name"

| inputlookup this_lookup | fields services AS Service_File_Name | format

Gives me the correct syntax to match the formatting of Splunk's logs.

Now I suppose the next question is: Can I put a * next to service names in the .CSV to represent a wildcard? Or does that need to be in the query string?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!