topic Re: Check if latest logs contain IOC in Splunk Search

Check if latest logs contain IOC

superuser88 — Thu, 24 Aug 2023 23:58:11 GMT

INDEX Name generated (10 million new records every day)
INDEX Fields username, secret, key

Lookup file secrets.csv with fields secret (128 bit string - 1 million static records)

I am creating a report to check if any of secret is found within the secrets.csv list and flag it.

index=generated [| inputlookup secrets.csv | fields secret] | table username, secret, key

How does the check for secret if exists in both index generated is validated in inputlookup in the search string?

bowesmana — Fri, 25 Aug 2023 01:17:08 GMT

You can't do that - because the subsearch will not return 1 million rows and it would take forever anyway.

index=generated | lookup secrets.csv secret OUTPUT secret as found | where isnotnull(found) | table username, secret, key

gcusello — Fri, 25 Aug 2023 08:06:08 GMT

if you want to search IOCs in all your raw log, you could try something like this:

index=generated [| inputlookup secrets.csv | rename secret AS query | fields query ] | table username key

In this way you perform a full text search on your raw log.

Ciao.

Giuseppe

bowesmana — Fri, 25 Aug 2023 08:18:13 GMT

You can't do a subsearch returning a million rows

ITWhisperer — Fri, 25 Aug 2023 09:14:39 GMT

Subsearches are limited to 50,000 events so if you still want to do it this way, you would have to split your lookup into about 20 different lookups.