Splunk Enterprise Security

Maximum Asset & Identity Lookup Size

malvidin
Communicator

What is the maximum recommended size for asset/identity lookups?

https://dev.splunk.com/enterprise/docs/developapps/enterprisesecurity/assetandidentityframework/ 

I've had issues with Splunk handling large numbers of assets and/or identities.  I increased the maximum bundle size to 4GB, but still had to distribute the entire huge bundle every time an identity changed.

Is there an option to use a KV store for assets & identities? Or a way to update them with a diff, rather than pushing the entire lookup?

Is there a memory requirement for a certain number of assets & identities? Or any related performance impact for having a large number of assets & identities?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Try to keep bundle size below 1GB. Beyond that you'll have problems.
Blacklist the A&I lookups from the bundle and push them to the indexers using a different method (scp via a cron job, for example).
---
If this reply helps you, Karma would be appreciated.

malvidin
Communicator

Thanks for the help.

With large A&I lookups, does Splunk provide memory recommendation for acceptable performance?

For example, if my asset list only contains ip, dns, priority, bunit,  andcategory, how many Class A networks can I put in the lookup if the networks are 50% allocated?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
The guidance is to keep bundle sizes below 1GB.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

I don't know if I can stay under that size, or even under 4GB, but I understand that is the recommended limit.

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Like I said in my first reply, if you keep large lookup files out of the bundle it will help keep the bundle size down.
---
If this reply helps you, Karma would be appreciated.
0 Karma

malvidin
Communicator

Thanks for clarifying that. Can KV lookups be distributed through different channels, like scp?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
KVStore collections are not included in the search bundle.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...