topic Re: Looking to see if a string is present in another search in Splunk Search

Looking to see if a string is present in another search

wweiland — Mon, 12 Oct 2015 19:06:14 GMT

I'm looking through DNS logs in one index. They are normal DNS logs, so they have the normal query containing the host+domain. I have threat data which just has the domain in another index.

index=main sourcetype=dns 
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

above is what i've come up with so far, but it doesn't seem to work. Am I looking at this wrong? I want to make sure that if the domain in the threat data exist at all in the dns domain data, then I get a alert.

Any suggestions?

Re: Looking to see if a string is present in another search

martin_mueller — Mon, 12 Oct 2015 19:56:22 GMT

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

Re: Looking to see if a string is present in another search

aljohnson_splun — Mon, 12 Oct 2015 20:00:19 GMT

When you are using map, it is better to have a deduplicated list of the values you want to map.

For example:

index=main sourcetype=dns
| dedup domain
| table domain
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

Another way you could do it would be using the set command:

| set intersect
[ index=main sourcetype=dns | fields - _* | fields domain]
[ index=sec u_type=domain u_status=Active | fields - _* | fields u_indicator]

Which I personally find a little clearer.

Please remember - subsearches have limits - please look at sideview's answer here for an example of using lookups.

Re: Looking to see if a string is present in another search

wweiland — Mon, 12 Oct 2015 20:39:07 GMT

Would this only trigger if the domain exactly matches the u_indicator?

Re: Looking to see if a string is present in another search

martin_mueller — Mon, 12 Oct 2015 20:48:03 GMT

Yes, so would your map search.

Re: Looking to see if a string is present in another search

wweiland — Mon, 12 Oct 2015 20:53:38 GMT

Can you think of a way I can write it so that if u_indicator is in any part of domain it would trigger? I was trying to think of how to use the if(match) but I can't seem to come up with the way.

Re: Looking to see if a string is present in another search

martin_mueller — Mon, 12 Oct 2015 21:35:28 GMT

Yeah, but it might be very slow:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain | eval domain = "*".domain."*"]

Re: Looking to see if a string is present in another search

martin_mueller — Mon, 12 Oct 2015 21:36:06 GMT

With the lookup-based approach, you can set the lookup to match using wildcards and include those asterisks in the lookup.

Re: Looking to see if a string is present in another search

wweiland — Mon, 12 Oct 2015 21:51:51 GMT

Thank you kind sir. That works like a charm.

Re: Looking to see if a string is present in another search

wweiland — Wed, 14 Oct 2015 15:27:07 GMT

It also occurred to me that I could output the threat data to outputcsv and then do the same query against inputcsv? Can I still do the eval if I do that?

Re: Looking to see if a string is present in another search

mreynov_splunk — Thu, 15 Oct 2015 01:59:12 GMT

evals are interpreted before lookups, but if they are in separate queries, should be no problem.

Re: Looking to see if a string is present in another search

wweiland — Thu, 15 Oct 2015 16:18:09 GMT

So the query works as written, but I ran into the problem that you suspected I would have. Trying to compare 50k threat domains to each record with a wildcard is taking 8 mins for a 15 min period. The only way I can think to resolve the problem would be to use a custom search command and drop it to python with dictionaries. Problem is, I don't control the Splunk back-end and can't add custom commands easily. Any suggestions on what I could do within native Splunk? Would lookup tables be faster than inputcsv?