Query to search for huge set of URLs

[rbathla]

I have close to 2000 URLs I want to search in one source. Is it possible to do it in one query by using lookup and what is the query?

[DalJeanis]

Somesoni2's answer is correct to what is technically possible.

The caution required is that, if you have a set of 2000 URLs that you are going to be searching for, though your entire source, then unless that is an indexed field, that search is likely to take slightly less than forever. Be sure to test with a lookup table of, say, 50 URLs across a limited time period where you know at least a few events should be, before submitting a search that might suck enough power to dim the lights.

[somesoni2 · Accepted solution]

You need to add those URLs in a csv file, say filterurl.csv with header as url, and add filterurl.csv as lookup table file in Splunk. Now try this,

if your data from that source has a field called url (same name as in lookup table) based on which you want to filter:

index=yourindex sourcetype=yoursourcetype source=yoursource [| inputlookup filterurl.csv | table url ]

if your data from that source has a field called myurl (different name as in lookup table) based on which you want to filter:

index=yourindex sourcetype=yoursourcetype source=yoursource [| inputlookup filterurl.csv | table url | rename url as myurl]

if your data doesn't have the field extracted and you want to do text based filter

index=yourindex sourcetype=yoursourcetype source=yoursource [| inputlookup filterurl.csv | table url | rename url as search]

[rbathla]

This was awesome. It worked. I am doing it in chunk of 500 and it is working.
Really appreciate the help always provided on the forum.

[Richfez]

Since that worked, could you please Accept somesoni2's answer? That will help future searchers find this easier and better!