Why am I getting "A lookup table used in a CIDR or WILDCARD definition exceeds the maximum allowable value" on my 3 Enterprise Security search heads?

[weicai88]

Hi Everyone:

I keep getting this error on my 3 Enterprise Security search heads:

msg="A lookup table used in a CIDR or WILDCARD definition exceeds the maximum allowable value" file="asn_by_cidr.csv" size="16360595" param="max_memtable_bytes" limit="10000000".

I am aware of the fix: https://answers.splunk.com/answers/152483/splunk-app-for-enterprise-security-where-to-change-the-set...,
but after I made the suggested change to all 3 search heads, the error keeps popping up.

I have verified with btool that the max_memtable_bytes limit has been set to 20000000:

./bin/splunk cmd btool --debug limits list |grep mem
/opt/splunk/etc/apps/tsp_esh_limits/default/limits.conf                max_memtable_bytes = 20000000

Any suggestions?

[jkat54]

Limits.conf is one file that is not passed to peers/indexers with the search bundle. You must put limits.conf on your peers/indexers too.

# limits.conf settings and DISTRIBUTED SEARCH
#   Unlike most settings which affect searches, limits.conf settings are not
#   provided by the search head to be used by the search peers.  This means that if
#   you need to alter search-affecting limits in a distributed environment, typically
#   you will need to modify these settings on the relevant peers and search head for
#   consistent results.

http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Limitsconf

[weicai88]

jkat54:

This sounds like what I need to do. I will test it tonight and let you know the result.

Thanks!
Wei

[dcarmack_splunk]

was a restart performed on the cluster?

[weicai88]

Yes, multiple times.

[dcarmack_splunk]

A few things:

1. The max_memtable_bytes parameter is set under the [lookup] stanza in your limits.conf?

2. Does asn_by_cidr.csv live in the tsp_esh_limits app, if not, is the app configured to share its configuration globally, as limits.conf is evaluated at the app/user level. If your lookup table lives in another app, and tsp_esh_limits does not share its configuration globally, then max_memtable_bytes = 20000000 will not apply.