Splunk Enterprise Security

Enterprise Security (ES) asset lookups failing with SRC and dest fields reporting “unknown-internal” and “unknown-external”

Splunk Employee
Splunk Employee

Running ES 5.1 on Splunk 7.1. The asset lookups have been working fine. This morning the SRC and dest fields display “unknown-internal” and “unknown-external” in place of the usual asset information. The lookups are populating normally, and nothing’s changed on the ES Search Head recently.

Digging deeper, the behavior makes it appear that the CIDR lookup asset_by_cidr.csv is happening before the string lookup assets_by_str.csv. But why would that be, and what would change the normal order-of-operations behavior for those lookups?

0 Karma

Splunk Employee
Splunk Employee

The size of the assets_by_str.csv lookup is exceeding the limits.conf setting max_memtable_bytes, and is being treated as a batched lookup. Splunk will manage lookups in-memory for CSVs less than 10MB (by default,) and index the rest as external batch based. The in-memory lookups get executed immediately, and order is consistent. For batched lookups, Splunk waits until a certain batch size is reached before performing a lookup operation. Due to this, there are higher chances of running into out-of-order conflicts when utilizing indexed lookups.

Raise the limits.conf setting max_memtable_bytes to a value larger than your assets_by_string.csv lookup on the SH and Indexers to eliminate the issue. Note: this will use more RAM on the hosts.

Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...