Splunk Search

Looking to see if a string is present in another search

Contributor

I'm looking through DNS logs in one index. They are normal DNS logs, so they have the normal query containing the host+domain. I have threat data which just has the domain in another index.

index=main sourcetype=dns 
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

above is what i've come up with so far, but it doesn't seem to work. Am I looking at this wrong? I want to make sure that if the domain in the threat data exist at all in the dns domain data, then I get a alert.

Any suggestions?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

When you are using map, it is better to have a deduplicated list of the values you want to map.

For example:

index=main sourcetype=dns
| dedup domain
| table domain
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

Another way you could do it would be using the set command:

| set intersect
[ index=main sourcetype=dns | fields - _* | fields domain]
[ index=sec u_type=domain u_status=Active | fields - _* | fields u_indicator]

Which I personally find a little clearer.


Please remember - subsearches have limits - please look at sideview's answer here for an example of using lookups.

0 Karma

SplunkTrust
SplunkTrust

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

View solution in original post

0 Karma

Contributor

Would this only trigger if the domain exactly matches the u_indicator?

0 Karma

SplunkTrust
SplunkTrust

Yes, so would your map search.

0 Karma

Contributor

Can you think of a way I can write it so that if u_indicator is in any part of domain it would trigger? I was trying to think of how to use the if(match) but I can't seem to come up with the way.

0 Karma

SplunkTrust
SplunkTrust

Yeah, but it might be very slow:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain | eval domain = "*".domain."*"]
0 Karma

Contributor

It also occurred to me that I could output the threat data to outputcsv and then do the same query against inputcsv? Can I still do the eval if I do that?

0 Karma

Splunk Employee
Splunk Employee

evals are interpreted before lookups, but if they are in separate queries, should be no problem.

0 Karma

Contributor

So the query works as written, but I ran into the problem that you suspected I would have. Trying to compare 50k threat domains to each record with a wildcard is taking 8 mins for a 15 min period. The only way I can think to resolve the problem would be to use a custom search command and drop it to python with dictionaries. Problem is, I don't control the Splunk back-end and can't add custom commands easily. Any suggestions on what I could do within native Splunk? Would lookup tables be faster than inputcsv?

0 Karma

Contributor

Thank you kind sir. That works like a charm.

0 Karma

SplunkTrust
SplunkTrust

With the lookup-based approach, you can set the lookup to match using wildcards and include those asterisks in the lookup.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!