Introduction
Raw telemetry is often incomplete. A log line might record an IP address, but what you really want in your observability platform is the owner, environment tag, or threat classification that goes with it. The OpenTelemetry Collector's lookup processor lets you close that gap at collection time — before the data ever reaches your backend.
This post walks through a concrete example: enriching logs that contain an IP address by looking it up in a CSV file.
What the lookup processor does
The lookup processor enriches telemetry signals (logs, traces, metrics) by:
Evaluating an OTTL expression to extract a key from each record.
Querying a lookup source (CSV, YAML, DNS, or a custom source) with that key.
Writing the result back as one or more new attributes on the same record or its resource.
The processor is part of opentelemetry-collector-contrib and is currently in development stability for logs, traces, and metrics.
Scenario: tagging logs by known IP address
Imagine your application writes JSON logs to disk. Your security and operations teams maintain a CSV file that maps known IP addresses to the owner, environment, and a risk label. You want every log from a known IP to carry those extra fields automatically — without changing the application.
The full pipeline is: filelog receiver → lookup processor → splunk_hec exporter.
The log file
Each line in /var/log/app/access.log is a JSON object:
{"timestamp":"2026-09-15T13:23:01Z","level":"info","message":"POST /api/orders","client_ip":"203.0.113.42","http_status_code":"201","duration_ms":42}
{"timestamp":"2026-09-15T13:23:05Z","level":"warn","message":"GET /admin","client_ip":"198.51.100.7","http_status_code":"403","duration_ms":5}
{"timestamp":"2026-09-15T13:23:09Z","level":"error","message":"GET /api/data","client_ip":"10.0.1.5","http_status_code":"503","duration_ms":1200}
{"timestamp":"2026-09-15T13:23:12Z","level":"info","message":"GET /healthz","client_ip":"192.0.2.99","http_status_code":"200","duration_ms":1}
The CSV files
/etc/otel/ip-metadata.csv — maps IP addresses to caller identity:
ip_address,owner,environment,risk
10.0.1.5,payments-service,production,low
10.0.2.31,load-tester,staging,medium
203.0.113.42,external-partner,production,low
198.51.100.7,unknown-scanner,external,high
/etc/otel/status_codes.csv — maps HTTP status codes to category and severity:
status_code,category,severity
200,success,info
201,success,info
301,redirect,info
400,client_error,warn
401,client_error,warn
403,client_error,warn
404,client_error,warn
500,server_error,error
502,server_error,error
503,server_error,critical
Collector configuration
Each processor instance has exactly one source , so the two CSV files require two named processor instances chained in the pipeline:
receivers:
filelog:
include: [/var/log/app/access.log]
start_at: beginning
operators:
- type: json_parser # parse each line as JSON
timestamp:
parse_from: attributes.timestamp
layout: "%Y-%m-%dT%H:%M:%SZ"
- type: move
from: attributes.client_ip
to: attributes["client.ip"]
- type: move
from: attributes.http_status_code
to: attributes["http.status_code"]
- type: move
from: attributes.message
to: body
processors:
lookup/ip:
source:
type: csv
path: /etc/otel/ip-metadata.csv
key_column: ip_address
reload_interval: 5m
lookups:
- key: log.attributes["client.ip"]
context: record # client.ip varies per record → record
attributes:
- source: owner
destination: client.owner
default: "unknown"
- source: environment
destination: client.environment
default: "unknown"
- source: risk
destination: client.risk
default: "unknown"
lookup/status:
source:
type: csv
path: /etc/otel/status_codes.csv
key_column: status_code
lookups:
- key: log.attributes["http.status_code"]
context: record # status code varies per request → record
attributes:
- source: category
destination: http.status_category
default: "unknown"
- source: severity
destination: http.status_severity
default: "info"
exporters:
splunk_hec:
token: "${SPLUNK_HEC_TOKEN}"
endpoint: "https://your-splunk-instance:8088/services/collector"
source: "otel"
sourcetype: "otel:logs"
index: "main"
service:
pipelines:
logs:
receivers: [filelog]
processors: [lookup/ip, lookup/status]
exporters: [splunk_hec]
The filelog receiver tails the log file and applies a chain of operators to each line:
json_parser — parses the raw JSON line and promotes its fields to log attributes, and sets the log record timestamp from the timestamp field.
move — renames client_ip → attributes["client.ip"] (dotted OTel convention the lookup processor expects).
move — renames http_status_code → attributes["http.status_code"] for the same reason.
move — promotes message to the log body, keeping attributes clean.
What happens at runtime
For each log record the processor chain:
lookup/ip reads client.ip , looks it up in the IP metadata table, and writes client.owner , client.environment , and client.risk to the log record.
lookup/status reads http.status_code , looks it up in the status codes table, and writes http.status_category and http.status_severity to the log record.
The third log line ( 503 from 10.0.1.5 ) arrives in Splunk with these log record attributes:
client.ip = "10.0.1.5"
client.owner = "payments-service"
client.environment = "production"
client.risk = "low"
http.status_code = "503"
http.status_category = "server_error"
http.status_severity = "critical"
The fourth line ( 192.0.2.99 , not in the IP CSV) gets defaults for the IP fields but is still enriched from the status code CSV:
client.ip = "192.0.2.99"
client.owner = "unknown"
client.environment = "unknown"
client.risk = "unknown"
http.status_code = "200"
http.status_category = "success"
http.status_severity = "info"
Scenario: tagging logs by host name
In the previous scenario client.ip varies per log record, so enrichment goes to the record. But what if you want to enrich logs based on which host they came from — your own servers? In that case the hostname is stable across all logs from that machine, making it a genuine resource-level attribute.
Imagine you have a fleet of servers and a CSV that maps each hostname to its team owner and environment. You want every log from a host to automatically carry that metadata, shared across all records from that host.
The CSV file
/etc/otel/host-inventory.csv :
host_name,team,environment
payments-prod-1,payments,production
payments-prod-2,payments,production
platform-staging-1,platform,staging
platform-staging-2,platform,staging
Collector configuration
The resource_detection processor automatically sets host.name as a resource attribute by reading it from the OS. The lookup processor then uses that resource attribute as the key:
receivers:
filelog:
include: [/var/log/app/*.log]
start_at: beginning
operators:
- type: json_parser
timestamp:
parse_from: attributes.timestamp
layout: "%Y-%m-%dT%H:%M:%SZ"
- type: move
from: attributes.message
to: body
processors:
resource_detection:
detectors: [system]
lookup/host:
source:
type: csv
path: /etc/otel/host-inventory.csv
key_column: host_name
reload_interval: 1h
lookups:
- key: resource.attributes["host.name"]
context: resource # host.name is a resource attribute → resource
attributes:
- source: team
destination: host.team
default: "unknown"
- source: environment
destination: deployment.environment
default: "unknown"
exporters:
splunk_hec:
token: "${SPLUNK_HEC_TOKEN}"
endpoint: "https://your-splunk-instance:8088/services/collector"
source: "otel"
sourcetype: "otel:logs"
index: "main"
service:
pipelines:
logs:
receivers: [filelog]
processors: [resource_detection, lookup/host]
exporters: [splunk_hec]
What happens at runtime
resource_detection runs first and sets host.name on the resource scope. Then lookup/host reads resource.attributes["host.name"] , looks it up in the host inventory, and writes host.team and deployment.environment back to the resource — once, shared by every log record from that host.
A log from payments-prod-1 arrives in Splunk with:
Resource attributes: host.name = "payments-prod-1" , host.team = "payments" , deployment.environment = "production"
All log records from that host carry the same resource attributes automatically.
This is the correct use of context: resource : the key ( host.name ) is already a resource attribute, and the enriched values are genuinely the same for every record from that source.
Record vs. resource: choosing the right context
The context field controls where enriched attributes land:
Record (default) — the attribute is written on each individual log record. Use this when the lookup key comes from a log attribute that varies per record, like client.ip or http.status_code .
Resource — the attribute is written on the resource scope, shared by all log records in that scope. Use this only when the lookup key comes from a resource attribute — something that is genuinely stable across all records from that source, like resource.attributes["host.name"] or resource.attributes["service.name"] .
The key rule: the context should match where the key comes from. If you key on log.attributes["client.ip"] but write to resource, all records in the same resource scope get the same enrichment — whichever IP happened to be processed first wins for the whole batch, which is incorrect.
The second scenario above (host inventory lookup) is the canonical correct use: host.name is already a resource attribute set by resource_detection , all records from that host share it, and the enrichment is legitimately the same for every record — so resource context is correct.
How the CSV source works
The CSV source loads the file during collector start-up and builds an in-memory map[string]any . With reload_interval set, it re-reads the file on that schedule — the previously loaded table stays in place if a reload fails, so a bad file never causes a data gap.
Because no value_column is specified in the examples above, the source stores the entire row as a map keyed by column name. Each source field in the attributes list picks one column out of that map.
If you only need a single enrichment field, point value_column at it and the source returns a scalar instead of a map:
source:
type: csv
path: /etc/otel/ip-risk.csv
key_column: ip_address
value_column: risk
lookups:
- key: log.attributes["client.ip"]
attributes:
- destination: client.risk
default: "unknown"
OTTL expressions on the key
The key field is a full OTTL value expression, not just a plain attribute path. You can apply converters before the lookup:
# Normalize the IP to lowercase before lookup (useful for IPv6)
- key: ConvertCase(log.attributes["client.ip"], "lower")
attributes:
- source: owner
destination: client.owner
Keeping the CSV up to date
The reload_interval option is the recommended way to keep the lookup table fresh without restarting the collector:
Write updates to a temporary file, then rename it into place. An atomic rename guarantees the collector never reads a half-written file during a reload.
Monitor the lookup_source_reloads and lookup_source_reload_failures counters the processor exposes. Alert on rising failure counts to detect stale tables.
Key takeaways
Capability
How it works
Match logs by IP
key: log.attributes["client.ip"]
Load enrichment data
source.type: csv with key_column
Add multiple fields per row
Omit value_column ; use source on each attribute mapping
Handle unknown IPs gracefully
default: on each attribute mapping
Keep the table current
reload_interval: 5m with atomic file writes
Multiple CSV sources
Use named instances: lookup/ip , lookup/status
Stable data → resource
context: resource on the lookup rule
Per-request data → record
context: record on the lookup rule
The lookup processor keeps your enrichment logic in the collector, out of your application code, and centrally configurable. A CSV on disk is often all you need — no side-car database required.
... View more