topic Maximum Asset & Identity Lookup Size in Splunk Enterprise Security

Maximum Asset & Identity Lookup Size

malvidin — Tue, 07 Jul 2020 15:48:09 GMT

What is the maximum recommended size for asset/identity lookups?

https://dev.splunk.com/enterprise/docs/developapps/enterprisesecurity/assetandidentityframework/

I've had issues with Splunk handling large numbers of assets and/or identities. I increased the maximum bundle size to 4GB, but still had to distribute the entire huge bundle every time an identity changed.

Is there an option to use a KV store for assets & identities? Or a way to update them with a diff, rather than pushing the entire lookup?

Is there a memory requirement for a certain number of assets & identities? Or any related performance impact for having a large number of assets & identities?

Re: Maximum Asset & Identity Lookup Size

richgalloway — Tue, 07 Jul 2020 17:02:06 GMT

Try to keep bundle size below 1GB. Beyond that you'll have problems.
Blacklist the A&I lookups from the bundle and push them to the indexers using a different method (scp via a cron job, for example).

Re: Maximum Asset & Identity Lookup Size

malvidin — Wed, 08 Jul 2020 08:32:02 GMT

Thanks for the help.

With large A&I lookups, does Splunk provide memory recommendation for acceptable performance?

For example, if my asset list only contains ip, dns, priority, bunit, andcategory, how many Class A networks can I put in the lookup if the networks are 50% allocated?

Re: Maximum Asset & Identity Lookup Size

richgalloway — Wed, 08 Jul 2020 12:32:11 GMT

The guidance is to keep bundle sizes below 1GB.

Re: Maximum Asset & Identity Lookup Size

malvidin — Sat, 11 Jul 2020 11:37:19 GMT

I don't know if I can stay under that size, or even under 4GB, but I understand that is the recommended limit.

Re: Maximum Asset & Identity Lookup Size

richgalloway — Sat, 11 Jul 2020 12:33:12 GMT

Like I said in my first reply, if you keep large lookup files out of the bundle it will help keep the bundle size down.

Re: Maximum Asset & Identity Lookup Size

malvidin — Sat, 11 Jul 2020 12:37:49 GMT

Thanks for clarifying that. Can KV lookups be distributed through different channels, like scp?

Re: Maximum Asset & Identity Lookup Size

richgalloway — Sat, 11 Jul 2020 16:49:36 GMT

KVStore collections are not included in the search bundle.