Splunk Enterprise Security Alexa Domain NOT lookup failing

[HannanPervez]

Hello,

I am trying to create alerts for all outbound DNS queries which do not match the top one million domains as per Alexa top 1 million which comes shipped with ES.

I am using the following search which I am fairly certain is correct as per Splunk base and previous examples of creating searches with lookups.

'index=externaldns | rex field=query "(?[A-z0-9]+.[A-z0-9]+$)"
| fields domain | search NOT [ | inputlookup alexa_by_str.csv ]| stats count by domain'

The result of this search was showing what looked to be the queries which DID match the top 1 million alexa domains. When looking into the job status of the search I saw an error message saying that the results were truncated to maxout 10000 (Splunks default) for searches: https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Limitsconf

I then changed both the maxout for the search AND subsearch to 12000000 in the limits.conf local file in my sh_cluster app to override the default and I am now receiving the following error message:

'[subsearch]: Search Processor: Subsearch produced 1002192 results, truncating to maxout 50000.'

Can someone tell me if I have changed the wrong config or if there is anything more than I should do to increase the maxout?

[DMohn · Accepted solution]

I would suggest to use a slightly different method here, as increasing the subsearch limits may clog your system. If I understand your use case, you want to get a list of all domains which are NOT included in the Alexa Top 1Mio and count these.

This can be done like:

'index=externaldns | rex field=query "(?[A-z0-9]+.[A-z0-9]+$)"
| fields domain 
| append [
    | inputlookup alexa_by_str.csv
    | eval is_alexa=1 ]
| stats count as count, sum(is_alexa) as is_alexa by domain
| where is_alexa=0
| table domain count

[HannanPervez]

That's worked perfectly DMohn. Many thanks!

[DMohn]

You're welcome, glad to be of assistance!

[HannanPervez]

Ah, just had another run through of the search there. The search works fine when setting the where = 1, as expected it displays all the dns queries made which match the alexa domain lookup. However, when setting the where = 0 there are no results found.

At first glace I'd thought that maybe it was possible that there no DNS queries made which aren't in the alexa lookup. However, I tested this by doing an nslookup on a domain NOT in the alexa domain lookup and then ran the search and still no results were found. The events for the nslookup are in the index but aren't showing up in the lookup search where alexa = 0.

Any thoughts?

[DMohn]

Try changing the where is_alexa=0 to where isnull(is_alexa)