INDEX Name generated (10 million new records every day)
INDEX Fields username, secret, key
Lookup file secrets.csv with fields secret (128 bit string - 1 million static records)
I am creating a report to check if any of secret is found within the secrets.csv list and flag it.
index=generated [| inputlookup secrets.csv | fields secret] | table username, secret, key
How does the check for secret if exists in both index generated is validated in inputlookup in the search string?
Subsearches are limited to 50,000 events so if you still want to do it this way, you would have to split your lookup into about 20 different lookups.
Hi @superuser88 ,
if you want to search IOCs in all your raw log, you could try something like this:
index=generated [| inputlookup secrets.csv | rename secret AS query | fields query ]
| table username key
In this way you perform a full text search on your raw log.
Ciao.
Giuseppe
You can't do a subsearch returning a million rows
You can't do that - because the subsearch will not return 1 million rows and it would take forever anyway.
index=generated
| lookup secrets.csv secret OUTPUT secret as found
| where isnotnull(found)
| table username, secret, key