Maximum Asset & Identity Lookup Size

[malvidin]

What is the maximum recommended size for asset/identity lookups?

https://dev.splunk.com/enterprise/docs/developapps/enterprisesecurity/assetandidentityframework/ 

I've had issues with Splunk handling large numbers of assets and/or identities.  I increased the maximum bundle size to 4GB, but still had to distribute the entire huge bundle every time an identity changed.

Is there an option to use a KV store for assets & identities? Or a way to update them with a diff, rather than pushing the entire lookup?

Is there a memory requirement for a certain number of assets & identities? Or any related performance impact for having a large number of assets & identities?

[richgalloway]

Try to keep bundle size below 1GB. Beyond that you'll have problems.
Blacklist the A&I lookups from the bundle and push them to the indexers using a different method (scp via a cron job, for example).

[malvidin]

Thanks for the help.

With large A&I lookups, does Splunk provide memory recommendation for acceptable performance?

For example, if my asset list only contains ip, dns, priority, bunit,  andcategory, how many Class A networks can I put in the lookup if the networks are 50% allocated?

[richgalloway · Accepted solution]

The guidance is to keep bundle sizes below 1GB.

[malvidin]

I don't know if I can stay under that size, or even under 4GB, but I understand that is the recommended limit.

[richgalloway]

Like I said in my first reply, if you keep large lookup files out of the bundle it will help keep the bundle size down.

[malvidin]

Thanks for clarifying that. Can KV lookups be distributed through different channels, like scp?

[richgalloway]

KVStore collections are not included in the search bundle.