Splunk Enterprise Security

Maximum Asset & Identity Lookup Size

malvidin
Communicator

What is the maximum recommended size for asset/identity lookups?

https://dev.splunk.com/enterprise/docs/developapps/enterprisesecurity/assetandidentityframework/ 

I've had issues with Splunk handling large numbers of assets and/or identities.  I increased the maximum bundle size to 4GB, but still had to distribute the entire huge bundle every time an identity changed.

Is there an option to use a KV store for assets & identities? Or a way to update them with a diff, rather than pushing the entire lookup?

Is there a memory requirement for a certain number of assets & identities? Or any related performance impact for having a large number of assets & identities?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Try to keep bundle size below 1GB. Beyond that you'll have problems.
Blacklist the A&I lookups from the bundle and push them to the indexers using a different method (scp via a cron job, for example).
---
If this reply helps you, Karma would be appreciated.

malvidin
Communicator

Thanks for the help.

With large A&I lookups, does Splunk provide memory recommendation for acceptable performance?

For example, if my asset list only contains ip, dns, priority, bunit,  andcategory, how many Class A networks can I put in the lookup if the networks are 50% allocated?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

I don't know if I can stay under that size, or even under 4GB, but I understand that is the recommended limit.

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Like I said in my first reply, if you keep large lookup files out of the bundle it will help keep the bundle size down.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

Thanks for clarifying that. Can KV lookups be distributed through different channels, like scp?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
KVStore collections are not included in the search bundle.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...