Check if latest logs contain IOC

[superuser88]

INDEX Name generated (10 million new records every day)
INDEX Fields username, secret, key 

Lookup file secrets.csv with fields secret (128 bit string - 1 million static records)

I am creating a report to check if any of secret is found within the secrets.csv list and flag it.

index=generated [| inputlookup secrets.csv | fields secret] | table username, secret, key

How does the check for secret if exists in both index generated is validated in inputlookup in the search string?

[ITWhisperer]

Subsearches are limited to 50,000 events so if you still want to do it this way, you would have to split your lookup into about 20 different lookups.

[gcusello]

Hi @superuser88 ,

if you want to search IOCs in all your raw log, you could try something like this:

index=generated [| inputlookup secrets.csv | rename secret AS query | fields query ] 
| table username key

In this way you perform a full text search on your raw log.

Ciao.

Giuseppe

[bowesmana]

You can't do a subsearch returning a million rows

[bowesmana]

You can't do that - because the subsearch will not return 1 million rows and it would take forever anyway.

index=generated 
| lookup secrets.csv secret OUTPUT secret as found
| where isnotnull(found)
| table username, secret, key