Looking to see if a string is present in another search

[wweiland]

I'm looking through DNS logs in one index. They are normal DNS logs, so they have the normal query containing the host+domain. I have threat data which just has the domain in another index.

index=main sourcetype=dns 
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

above is what i've come up with so far, but it doesn't seem to work. Am I looking at this wrong? I want to make sure that if the domain in the threat data exist at all in the dns domain data, then I get a alert.

Any suggestions?

[aljohnson_splun]

When you are using map, it is better to have a deduplicated list of the values you want to map.

For example:

index=main sourcetype=dns
| dedup domain
| table domain
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

Another way you could do it would be using the set command:

| set intersect
[ index=main sourcetype=dns | fields - _* | fields domain]
[ index=sec u_type=domain u_status=Active | fields - _* | fields u_indicator]

Which I personally find a little clearer.

---

Please remember - subsearches have limits - please look at sideview's answer here for an example of using lookups.

[martin_mueller · Accepted solution]

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

[wweiland]

Would this only trigger if the domain exactly matches the u_indicator?

[martin_mueller]

Yes, so would your map search.

[wweiland]

Can you think of a way I can write it so that if u_indicator is in any part of domain it would trigger? I was trying to think of how to use the if(match) but I can't seem to come up with the way.

[martin_mueller]

Yeah, but it might be very slow:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain | eval domain = "*".domain."*"]

[wweiland]

It also occurred to me that I could output the threat data to outputcsv and then do the same query against inputcsv? Can I still do the eval if I do that?

[mreynov_splunk]

evals are interpreted before lookups, but if they are in separate queries, should be no problem.

[wweiland]

So the query works as written, but I ran into the problem that you suspected I would have. Trying to compare 50k threat domains to each record with a wildcard is taking 8 mins for a 15 min period. The only way I can think to resolve the problem would be to use a custom search command and drop it to python with dictionaries. Problem is, I don't control the Splunk back-end and can't add custom commands easily. Any suggestions on what I could do within native Splunk? Would lookup tables be faster than inputcsv?

[wweiland]

Thank you kind sir. That works like a charm.

[martin_mueller]

With the lookup-based approach, you can set the lookup to match using wildcards and include those asterisks in the lookup.