Splunk Search

Filtering queries by large lookup

Path Finder

I have a large lookup full of bad domains. The lookup is simply a domain name per line. I would like to search through a section of logs and return events where the bad domain exists. My domain list looks like

baddomain
baddomain1.com
baddomain2.com
...

And the query I'm attempting to write looks like

index=myindex | append [ inputlookup domains.csv | fields baddomain ] | where domain=baddomain | ...

If I run just the inputlookup portion of the query I'm seeing my results, but they do not seem to be getting appended.

Thanks, Kevin

Tags (2)
2 Solutions

Splunk Employee
Splunk Employee

How about:

index=myindex [ inputlookup domains.csv | rename baddomain as domain | fields baddomain]

This should be equivalent to:

index=myindex (domain=badguy.com OR domain=donkey.com ...)

View solution in original post

Path Finder

So we worked the issue. The confusion was in the distributed search aspect.

We originally had the lookup file pushed out using the deployment server. So each indexer had the file in (app)/local/lookups. The search head did not have the lookup file.

It appears (please correct if this is wrong) that when using inputlookup, the lookup file need only be on the search head.

We were confusing this with regular lookups, where 'local=t' can be specified.

View solution in original post

Path Finder

So we worked the issue. The confusion was in the distributed search aspect.

We originally had the lookup file pushed out using the deployment server. So each indexer had the file in (app)/local/lookups. The search head did not have the lookup file.

It appears (please correct if this is wrong) that when using inputlookup, the lookup file need only be on the search head.

We were confusing this with regular lookups, where 'local=t' can be specified.

View solution in original post

Path Finder

GO TEAM NOVII!!

0 Karma

Path Finder

you could also try:

[] LOOKUP-baddomain = ..... OUTPUTNEW ...

in your props.conf using the true and is_bad field.

with associated entry in transforms.conf:

[baddomain] default_match = no_match filename = lookup_baddomains.csv max_matches = 1 min_matches = 1

This way the search would be:

index=myindex is_bad=true

as the lookup will be appended automatically

Builder

I typically use "lookup" instead of inputlookup along with a boolean column:

## baddomains.csv

domain,is_bad
bad.domain.com,true

index=myindex | lookup domains.csv domain OUTPUT is_bad | search is_bad=true

Splunk Employee
Splunk Employee

How about:

index=myindex [ inputlookup domains.csv | rename baddomain as domain | fields baddomain]

This should be equivalent to:

index=myindex (domain=badguy.com OR domain=donkey.com ...)

View solution in original post

Path Finder

So I've become a liar. 2 issues.

  1. In a distributed setup, it will not work unless the lookup is on the search head. Is there a way to tell it to look locally?
  2. When I did put this file on the search head I got the error "Error in 'UnifiedSearch': Unable to parse 'The specified file is too large..."
0 Karma

Path Finder

Did I ever tell you you're my hero?
You're everything, everything I wish I could be.

Thanks dude, works swell!

0 Karma