Splunk Search

lookup table slows down the search query

splunker12er
Motivator

I have a lookup table blacklist.csv , which has blacklisted src & dest IPs. Using the below search query , I am listing the events containing blacklisted IPs

index=* sourcetype=pan* [|inputlookup blacklist.csv | fields dest_ip] OR [|inputlookup blacklist.csv | fields src_ip]  | table dest_ip,src_ip,status,etc..

My lookup table has 10,000 records, which makes my search slow down. Is there a way optimize the query ?
or any other mechanism to speed up the search ?

0 Karma

lmyrefelt
Builder

You could index the csv file into an index, that i am sure will speed up your search.

If you have a distributed environment and your lookup file is BIG , you also add local=t .

Like;

[|inputlookup local=t blacklist.csv

That helps to speed it up sometimes .

0 Karma

somesoni2
Revered Legend

See if this is faster than your current query.

index=* sourcetype=pan* [|inputlookup blacklist.csv | table dest_ip src_ip | format | format "" "" "OR" "" "OR" ""]  | table dest_ip,src_ip,status,etc..
0 Karma

splunker12er
Motivator

No there are no duplicate entries in the lookup table.
I tried dedup but it takes the same time..
For a very long time , still the search query is in parsing state.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Are there any duplicates in the list? As a first step, add a dedup dest_ip and dedup src_ip to each.

0 Karma
Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Building Momentum: Splunk Developer Program at .conf25

At Splunk, developers are at the heart of innovation. That’s why this year at .conf25, we officially launched ...