Splunk Enterprise Security

Enterprise Security (ES) asset lookups failing with SRC and dest fields reporting “unknown-internal” and “unknown-external”

ekost
Splunk Employee
Splunk Employee

Running ES 5.1 on Splunk 7.1. The asset lookups have been working fine. This morning the SRC and dest fields display “unknown-internal” and “unknown-external” in place of the usual asset information. The lookups are populating normally, and nothing’s changed on the ES Search Head recently.

Digging deeper, the behavior makes it appear that the CIDR lookup asset_by_cidr.csv is happening before the string lookup assets_by_str.csv. But why would that be, and what would change the normal order-of-operations behavior for those lookups?

0 Karma

ekost
Splunk Employee
Splunk Employee

The size of the assets_by_str.csv lookup is exceeding the limits.conf setting max_memtable_bytes, and is being treated as a batched lookup. Splunk will manage lookups in-memory for CSVs less than 10MB (by default,) and index the rest as external batch based. The in-memory lookups get executed immediately, and order is consistent. For batched lookups, Splunk waits until a certain batch size is reached before performing a lookup operation. Due to this, there are higher chances of running into out-of-order conflicts when utilizing indexed lookups.

Raise the limits.conf setting max_memtable_bytes to a value larger than your assets_by_string.csv lookup on the SH and Indexers to eliminate the issue. Note: this will use more RAM on the hosts.

Get Updates on the Splunk Community!

Thanks for the Memories! Splunk University, .conf24, and Community Connections

Thank you to everyone in the Splunk Community who joined us for .conf24 – starting with Splunk University and ...

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...