Splunk Enterprise Security

Maximum Asset & Identity Lookup Size

malvidin
Communicator

What is the maximum recommended size for asset/identity lookups?

https://dev.splunk.com/enterprise/docs/developapps/enterprisesecurity/assetandidentityframework/ 

I've had issues with Splunk handling large numbers of assets and/or identities.  I increased the maximum bundle size to 4GB, but still had to distribute the entire huge bundle every time an identity changed.

Is there an option to use a KV store for assets & identities? Or a way to update them with a diff, rather than pushing the entire lookup?

Is there a memory requirement for a certain number of assets & identities? Or any related performance impact for having a large number of assets & identities?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Try to keep bundle size below 1GB. Beyond that you'll have problems.
Blacklist the A&I lookups from the bundle and push them to the indexers using a different method (scp via a cron job, for example).
---
If this reply helps you, Karma would be appreciated.

malvidin
Communicator

Thanks for the help.

With large A&I lookups, does Splunk provide memory recommendation for acceptable performance?

For example, if my asset list only contains ip, dns, priority, bunit,  andcategory, how many Class A networks can I put in the lookup if the networks are 50% allocated?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

I don't know if I can stay under that size, or even under 4GB, but I understand that is the recommended limit.

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Like I said in my first reply, if you keep large lookup files out of the bundle it will help keep the bundle size down.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

Thanks for clarifying that. Can KV lookups be distributed through different channels, like scp?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
KVStore collections are not included in the search bundle.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...