Splunk Asset Filtering and Enrichment (at scale)

[lazysecurity]

Hi,

I'm still fairly new to Splunk (come from an ArcSight background) so apologies if this is a silly question.

Background

I'm trying to build 'non-signature' use cases (e.g. Service Installed or Unknown Process) which rely on knowing what is normal (with a whitelist) and alerting anything else.
When I've done this in the past it is normally sensible to start with critical assets and gradually increase host coverage over time. This allows the SOC to baseline what is normal for a subset of the most important hosts. Once you've 'learnt' what is normal for these hosts you expand coverage. This is pretty straightforward in ArcSight using an Asset Model. It also allows the events to be enriched with the asset criticality, Line of Business, etc.

Which brings me to the problem..

I'm trying to limit coverage & enrich events for a large environment in Splunk using lookups. Essentially we go to our lookup and for specific apps retrieve the hosts that need to be covered. Given the size of the environment, retrieving and applying this to the search is taking forever. For example if we apply logic across all hosts the search might take 10-15 s. When apply the same logic but try to limit to the subset of hosts it takes 5-6 min which is not going to fly given the number of searches we will need to run for different use cases.

I understand KV might be faster so are going to try that but has anyone else tried doing asset coverage and/or enrichment on Splunk? I've looked at http://docs.splunk.com/Documentation/ES/4.7.1/Admin/Addassetandidentitydata but it seems to be just applying an automatic lookup at search time which I would imagine would have the same performance hit.

Thanks in advance

[jplumsdaine22]

One way to approach this is to only apply enrichment pre or post search. For example if you have a lookup table with asset data then your search becomes (as an example)

[ | inputlookup assets.csv | fields host ] .... your search

OR

... your search |  lookup assets host OUTPUT some_enrichment_field

The gist of this is to reduce the lookups to a small a set as possible. Avoid for example trying to enrich millions of events - there's unlikely to be a good reason as you should be able to filter the events before enrichment. If you have some examples we can help you improve your searches.

That said I don't know what kind of scale you're talking about. If you're indexing millions of events a second you may very well be running into limitations.