Splunk Enterprise Security

How to resolve replication errors on knowledge bundle size over 200MB due to Splunk Enterprise Security identities and assets?

tkiss
Path Finder

Hi,

I'm looking for some answer and suggestion how I could decrease/workaround the knowledge bundle replication errors we're seeing in our environment which is a multi site indexer clustering with simple search head (no clustering, pooling, etc) that is installed with Enterprise Security.

Identities_expanded.csv and assets_by_asn.csv are HUGE, I mean just these 2 files are 150MB+. This makes the knowledge bundle huge as well which we need to replicate between Europe and US across the WAN where our 2 Splunk sites reside. 4-4 indexers / site.

If I blacklist the csv files, then I'm full of errors during search - complaining about missing lookups.

Do you have any workaround/suggestion how we could tackle this problem?

Thanks
tkiss

0 Karma

AndySplunks
Communicator

Did you ever resolve this issue?

0 Karma

goodsellt
Contributor

Are you getting timeout errors? You could try bumping up that timeout clock for these.

A lot of apps with large lookups also recommend that you put a copy of them onto the indexers themselves, however since this is probably not recommended for ES and generally overkill, you could try making your own custom app containing the CSV files and the transforms.conf lookup stanzas then deploy it to your index clusters and see if that helps, since it should be able to detect and use the lookup files locally at that point.

0 Karma

tkiss
Path Finder

Thanks, however timeout is already increased to 60000 (seconds) - didn't really help. Timeouts and especially delays/lags happen upon searches very frequently. Not to mention the network folks are pretty upset because we're putting unnecessary load on the WAN connection between US and EU (4x200MB transmitted just for the knowledge bundle replication).

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...