- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm trying to use wildcards in a lookup file and am not able to get them working. I have referenced other posted answers but am not having success. Steps I have taken:
Created a lookup file called 'dt_s.csv' using the web interface by uploading the following content:
cs_host, is_suspicious
www.google.com, yes
www.*, yesCreated a lookup definition called 'dt' using the web interface, based off dt_s.csv
Our administrator added the below to transforms.conf on the indexers
[dt]
filename = dt_s.csv
match_type = WILDCARD(cs_host)
When I run a search such as '-search- | lookup dt cs_host | head 50 | fields cs_host, is_suspicious' I only get results for www.google.com and nothing for any other www.* entries.
What are we doing wrong? Is there any other step-by-step official documentation on how to set this up? Thank you.
Answer in comments below: In a pre v6.6 deployment with indexers separate from search head, the [dt] section above has to be added to the local transforms.conf on indexers AND the search head.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tested this and got successful results with
index=*
| head 1
| eval cs_host="www.foo.com"
| lookup dt cs_host | fields cs_host, is_suspicious
I get the same results with www.google.com and www.foo.com (i.e. is_suspicious=yes).
2 things to consider:
1) Look at app context and permissions on the lookup knowledge objects. I doubt this is your issue since it works for the google domain.
2) I was able to do this all from the GUI without having to edit my transforms manually by configuring "advanced" options under the lookup definition, and using WILDCARD(cs_host) as my match type.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I tested this and got successful results with
index=*
| head 1
| eval cs_host="www.foo.com"
| lookup dt cs_host | fields cs_host, is_suspicious
I get the same results with www.google.com and www.foo.com (i.e. is_suspicious=yes).
2 things to consider:
1) Look at app context and permissions on the lookup knowledge objects. I doubt this is your issue since it works for the google domain.
2) I was able to do this all from the GUI without having to edit my transforms manually by configuring "advanced" options under the lookup definition, and using WILDCARD(cs_host) as my match type.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Andrew. We're still on 6.5.4 which does not have that option in the web interface, and we're some months away for a 6.6.x upgrade. The administrator put the configuration block in /opt/splunk/etc/system/local/transforms.conf on the indexers only, not on the search head. Do you think it needs to be added to the search head as well? Does splunk need to be reloaded for the changes to take affect?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes - lookups are a search-time transform, not index-time. I'm pretty sure transforms.conf must be on the search head for this to work.
https://docs.splunk.com/Documentation/Splunk/6.5.3/Knowledge/Searchtimeoperationssequence
I'm not sure if it will require reloading, but you can do this without necessarily restarting the search head by hitting the REST endpoint. See here:
https://answers.splunk.com/answers/102568/reload-transforms-conf-without-restarting-splunk.html
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Andrew, that did it. We added transforms.conf section to the search head as well and now it's working.
So in short, in a pre v6.6 deployment with indexers separate from search head, the [dt] section above has to be added to the local transforms.conf on indexers AND the search head.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Adauria,
I made the change in the advanced options, adding WILDCARD(process), and then ran the following search where I'm trying to match on executables other than the two paths, but I'm seeing all process file names and not just defrag.exe that I copied to the desktop and executed.
sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational EventCode=1 Image!=Windows\\System32 Image!=Windows\\SysWOW64 | eval process=lower(process) | lookup isWindowsSystemFile_lookup process | search systemFile=true | table _time dest host user process Image
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Figured out what the issue was - had the modify the search to as follows:
sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational EventCode=1 NOT ("Windows\\System32" OR "Windows\\SysWOW64") | eval process=lower(process) | lookup isWindowsSystemFile_lookup process | search systemFile=true | table _time dest host user process Image
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What does your lookup csv look like? Maybe your wildcards are too broad?
|inputlookup isWindowsSystemFile_lookup
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Lookup file is as follows:
process systemFile
arp.exe* true
adaptertroubleshooter.exe* true
applicationframehost.exe* true
atbroker.exe* true
authhost.exe* true
autoworkplace.exe* true
etc etc
Thx
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have done it according this answer and it works
https://answers.splunk.com/answers/52580/can-we-use-wildcard-characters-in-a-lookup-table.html
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Including the props.conf changes? I was under the impression props.conf should only be necessary if we want the lookup to be automatic, which I definitely do NOT in this case.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We used it only with props.conf
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't understand. You only made the props.conf changes and not the transforms.conf changes?
Can you share your lookup name and the what you put in props.conf as an example?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
to clarify we done it with transforms.conf and props.conf as we used automatic lookups and did not tested it with transforms.conf only. So our configs was same as in example:
props.conf
[yoursourcetype]
LOOKUP-user = userlookup user OUTPUT username
transforms.conf
[userlookup]
filename = userlookup.csv
match_type = WILDCARD(user)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks. Did you deploy the transforms.conf changes on the search head or on the indexers?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
transforms.conf seems correct. Problem with your search query