Hello,
After some time spent Googling/Splunking yesterday, I could not find a unique solution to my problem.
Goal: I have a list of services (splunkd.exe, splunkweb.exe, svchost.exe, etc) that I want to exclude from a search without having a query string that is 25 lines. I want to read the exclusion list from a CSV.
Query pseudocode: Search index to see if any services have been installed on any systems. EXCLUDE services from
Good things: If I just have | inputlookup this_lookup | fields services, then I can see all of my values of that field in a table in splunk.
Bad things: If I say NOT | inputlookup this_lookup | fields services | It doesn't recognize the match between the values in the CSV and the service_file_names in the logs, returns ALL results.
Bottom line (& at the end of the day? ;)): I noticed that in the events, the Service_File_Name is a full path, i.e. C:\Win\temp\this.sys, sometimes with quotes, sometimes with %sys%. Is it possible that the formatting I have stored these service file names in is not appropriate? It works as a query to say NOT (this.sys OR mcafee.exe), so I used the same formatting for those service names in my CSV.
Looking for direction, thank you.
P.S. I've looked at almost all of the inputlookup questions (and read the documentation) and haven't found my solution.
I suspect something is wrongly formatted in your lookup, because the inputlookup
approach you've used is what I was just about to suggest.
<any other search parameters> NOT [| inputlookup this_lookup | fields services]
To look at how Splunk sees what comes from the lookup, you could run the subsearch as its own search and add format
to the end:
| inputlookup this_lookup | fields services | format
This is exactly what the subsearch will get substituted with, so if you take the search string that you get from format
and it doesn't work correctly when you enter it directly into the search field, then there's your problem. Just note that you need to think about negating this with an initial NOT to really try the same thing as in your first search.
You are on the right track. Perhaps a slightly different approach will help. Why not create a blacklist of services in your lookup table. That means your CSV is named "service_black_list.csv" an it has content like this:
service_name, exclude
splunkd.exe, true
splunkweb.exe, true
svchost,exe, true
When you make the association with the lookup, you should ensure that the default value is false.
props.conf
[WinEventLog:Security]
LOOKUP-service_black_list = service_black_list service_name AS service_name OUTPUTNEW
transforms.conf
[service_black_list]
filename = service_black_list.csv
min_matches = 1
default_match = false
The upshot benefit then is that your seach becomes simple, like this:
sourcetype="WinEventLog:Security" service_name="*" exclude="false"
This will list only those services that are _not listed in the CSV.
I hope this helps.
Yes, if the Service_File_Name is a full path, your lookup table should contain the full path, too. There is a way around this, but it is just much easier to set up the lookup table for an exact match.
For this solution, put a second field in your lookup table. I called it "Matched". For each service, I put a value of "yes". So the table looks like
Service_File_Name,Matched
servicepath1,yes
servicepath2,yes
...
In my example, I have called the lookup service_lookup
. You should also set the Advanced Options on the lookup, so that the min and max matches = 1 and the default value is NoMatch.
yoursearchhere
| lookup service_lookup Service_File_Name OUTPUT Matched
| where Matched="NoMatch"
It would probably be more efficient to do the search as you originally imagined it, but this will work.This search will eliminate events with a Service_File_Name that appears in the lookup table.
I suspect something is wrongly formatted in your lookup, because the inputlookup
approach you've used is what I was just about to suggest.
<any other search parameters> NOT [| inputlookup this_lookup | fields services]
To look at how Splunk sees what comes from the lookup, you could run the subsearch as its own search and add format
to the end:
| inputlookup this_lookup | fields services | format
This is exactly what the subsearch will get substituted with, so if you take the search string that you get from format
and it doesn't work correctly when you enter it directly into the search field, then there's your problem. Just note that you need to think about negating this with an initial NOT to really try the same thing as in your first search.
lguinn,
It is working. Simple, but, working. I am very new to Splunk and want to get into good habits.
What is the benefit to using output?
Yes, you can put a * in the CSV file - if you are using it this way. Glad to see that this option is working for you as well.
Ayn,
Thanks for the response.
I did as you advised and I noticed that it is searching for "services=this.exe" which is obviously not the same as Splunk's "Service_File_Name"
| inputlookup this_lookup | fields services AS Service_File_Name | format
Gives me the correct syntax to match the formatting of Splunk's logs.
Now I suppose the next question is: Can I put a * next to service names in the .CSV to represent a wildcard? Or does that need to be in the query string?