Activity Feed
- Karma Re: Wildcards in lookup file for adauria_splunk. 06-05-2020 12:48 AM
- Karma Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error? for DalJeanis. 06-05-2020 12:48 AM
- Got Karma for Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?. 06-05-2020 12:48 AM
- Karma Re: Apache access_combined with X-Forwarded-For instead of host for jgoddard. 06-05-2020 12:46 AM
- Karma Re: How do I find all duplicate events? for dmaislin_splunk. 06-05-2020 12:45 AM
- Posted Re: Disable automatic lookups for a search on Splunk Search. 11-06-2019 07:41 AM
- Posted Re: Disable automatic lookups for a search on Splunk Search. 10-29-2019 08:04 AM
- Posted Re: Disable automatic lookups for a search on Splunk Search. 10-29-2019 07:54 AM
- Posted Disable automatic lookups for a search on Splunk Search. 10-25-2019 09:44 AM
- Tagged Disable automatic lookups for a search on Splunk Search. 10-25-2019 09:44 AM
- Posted Re: Apache access_combined with X-Forwarded-For instead of host on Getting Data In. 11-17-2017 11:14 AM
- Posted Re: Wildcards in lookup file on Splunk Search. 07-13-2017 02:37 PM
- Posted Re: Wildcards in lookup file on Splunk Search. 07-12-2017 08:44 AM
- Posted Re: Wildcards in lookup file on Splunk Search. 06-30-2017 12:14 PM
- Posted Re: Wildcards in lookup file on Splunk Search. 06-30-2017 07:19 AM
- Posted Re: Wildcards in lookup file on Splunk Search. 06-30-2017 06:57 AM
- Posted Wildcards in lookup file on Splunk Search. 06-30-2017 05:38 AM
- Tagged Wildcards in lookup file on Splunk Search. 06-30-2017 05:38 AM
- Tagged Wildcards in lookup file on Splunk Search. 06-30-2017 05:38 AM
- Tagged Wildcards in lookup file on Splunk Search. 06-30-2017 05:38 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 |
11-06-2019
07:41 AM
Perfect, that's exactly what I'm looking for! kv was not in my regular arsenal of commands. Simple and dynamic, thank you.
... View more
10-29-2019
08:04 AM
Thanks. Almost all of the lookups come from TAs like the TA for Windows or Identity Management. They're doing things like pulling out any 'user' field regardless of source type and enriching with LDAP information.
Other examples are the built in date_ fields. I don't think those are lookups, are they? Maybe I'm asking with the wrong term.
What I'm really looking for is a search command that will output only the fields that were extracted from the log message itself - no enrichments, no lookups, no built-ins. Does such a thing exist?
... View more
10-29-2019
07:54 AM
That's an idea. I don't think our admins would go for that though. Many of the lookups come from TAs.
... View more
10-25-2019
09:44 AM
I am using a summary index where the events being added to it contain different types of data, and therefore have different kinds and numbers of fields. I want to run a generic search against this summary index for when type=alert and have an email generated containing all the fields for that event.
The problem is others using this search head have defined many different automatic lookups that sometimes match my fields. I do not want the alert email to contain any of the automatic lookup output.
How can I disable all automatic lookups for just this search so my email isn't filled with junk? I found this previous question and understand the on up-voted response, but the question is not actually answered - https://answers.splunk.com/answers/113653/ignore-automatic-lookup-just-for-a-search.html
I contemplated doing "|fields -" but there are over 150 auto lookup fields in some cases... Is there a way to disable lookups for a specific eventtype, in which case I can define one for this summary index?
Thanks in advance.
... View more
- Tags:
- splunk-enterprise
11-17-2017
11:14 AM
This answer pointed me in the right direction but it was missing a piece of the puzzle - changes to props.conf. Here are all the configuration changes that had to be made to turn clientip into a multivalued field and correctly parse the X_Forwarded_For IPs:
transforms.conf - modified. The new regex is highlighted
[access-extractions]
REGEX = ^**(?<all_xff_ip>(([.\d]+|[a-fA-f0-9\:\.]+|-|localhost)(?:,\s)?)+)**\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]]
# new section
[clientip]
SOURCE_KEY=all_xff_ip
REGEX = (?P<clientip>[.:\d]+|[a-fA-f0-9\:.]+|-|localhost)
MV_ADD = true
props.conf
# new section
[access_combined]
REPORT-access_combined_clientip = clientip
... View more
07-13-2017
02:37 PM
Andrew, that did it. We added transforms.conf section to the search head as well and now it's working.
So in short, in a pre v6.6 deployment with indexers separate from search head, the [dt] section above has to be added to the local transforms.conf on indexers AND the search head.
... View more
07-12-2017
08:44 AM
Thanks Andrew. We're still on 6.5.4 which does not have that option in the web interface, and we're some months away for a 6.6.x upgrade. The administrator put the configuration block in /opt/splunk/etc/system/local/transforms.conf on the indexers only, not on the search head. Do you think it needs to be added to the search head as well? Does splunk need to be reloaded for the changes to take affect?
... View more
06-30-2017
12:14 PM
Thanks. Did you deploy the transforms.conf changes on the search head or on the indexers?
... View more
06-30-2017
07:19 AM
I don't understand. You only made the props.conf changes and not the transforms.conf changes?
Can you share your lookup name and the what you put in props.conf as an example?
... View more
06-30-2017
06:57 AM
Including the props.conf changes? I was under the impression props.conf should only be necessary if we want the lookup to be automatic, which I definitely do NOT in this case.
... View more
06-30-2017
05:38 AM
I'm trying to use wildcards in a lookup file and am not able to get them working. I have referenced other posted answers but am not having success. Steps I have taken:
Created a lookup file called 'dt_s.csv' using the web interface by uploading the following content:
cs_host, is_suspicious
www.google.com, yes
www.*, yes
Created a lookup definition called 'dt' using the web interface, based off dt_s.csv
Our administrator added the below to transforms.conf on the indexers
[dt]
filename = dt_s.csv
match_type = WILDCARD(cs_host)
When I run a search such as '-search- | lookup dt cs_host | head 50 | fields cs_host, is_suspicious' I only get results for www.google.com and nothing for any other www.* entries.
What are we doing wrong? Is there any other step-by-step official documentation on how to set this up? Thank you.
Answer in comments below: In a pre v6.6 deployment with indexers separate from search head, the [dt] section above has to be added to the local transforms.conf on indexers AND the search head.
... View more
06-22-2017
08:43 AM
1 Karma
The root cause ended up being a custom app with very long file paths caused the knowledge bundle to fail to replicate. Once the app was removed, the knowledge bundle began replicating again and lookups started working.
The is apparently a known issue with 6.5.1 and has been resolved in some later version.
... View more
05-15-2017
09:52 AM
6.5.1. Search head is standalone. There are multiple indexers.
... View more
05-12-2017
10:14 AM
Here you are. Hostname and index name removed for privacy.
inputlookup:
<hostname>/en-US/app/search/search?q=%7Cinputlookup%20dt1&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=statistics&sid=1494609014.208099&display.page.search.tab=statistics
index= |lookup dt1 cs_host
<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609057.208159
index= |lookup local=true dt1 cs_host
<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20local%3Dtrue%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609075.208172
... View more
05-12-2017
05:15 AM
We're not able to find a distsearch.conf file in the search app directory. The query does succeed using local=true.
... View more
05-10-2017
11:20 AM
Searches with lookups are failing in our environment. I have created a lookup file called dt1.csv and a lookup definition called dt1. Both the file and the definition have read and write permissions for all users in the search app.
This works successfully and shows the contents of the lookup:
|inputlookup dt1
However, using the lookup as part of a query such as:
<search> |lookup dt1 cs_host
Fails with these errors:
[indexer1] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.
[indexer2] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.
Every indexer errors out with that message. This happens regardless of the app the lookup is created in, and regardless of the user that creates the lookup.
Any ideas on what would cause lookups to fail with these errors? We're on Splunk 6.5.1.
... View more
