topic Re: Ingest time lookup to add fields does not work in Getting Data In

Ingest time lookup to add fields does not work

patelmc — Tue, 09 Nov 2021 00:51:21 GMT

I need to use federated search which does not support search time lookup at this time in splunk 8.2.2.1.

I came across splunk doc to add fields at ingest time (index time) based on ingest time lookup.

https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/IngestLookups

What I am trying to do is during event ingestion I am looking for value of field "application" and match that with the CSV file as shown below and trying to add fields APP and COMP based on application value.

e.g. if incoming event has application=Linux add APP field with value 9001 and COMP field as 8001.

But it does not work. Please help.

Here are the following files I created as documented.

more /opt/splunk/etc/system/lookups/APP_COMP.csv
application,APP,COMP
Linux,9001,8001
Console,9002,8002
Windows,9003,8003

more /opt/splunk/etc/system/local/props.conf
[access_combine_wcookie]
TRANSFORMS = Active_Events

/opt/splunk/etc/system/local/transforms.conf
[Active_Events]
INGEST_EVAL= APPCOMP=lookup("APP_COMP.csv", json_object("application", application), json_array("APP", "COMP"))

more /opt/splunk/etc/system/local/fields.conf
[APP]
INDEXED = True
[COMP]
INDEXED = True

Re: Ingest time lookup to add fields does not work

victor1004k — Mon, 26 May 2025 09:05:43 GMT

[Solution]

@patelmc You can achieve the desired result by modifying the content below slightly.

1. /opt/splunk/etc/apps/myapp/local/transforms.conf

[Active_Events] INGEST_EVAL= application=replace(_raw, ".*application=(\w+).*", "\1"), APP=json_extract(lookup("APP_COMP.csv", json_object("application", application), json_array("APP")),"APP"), COMP=json_extract(lookup("APP_COMP.csv", json_object("application", application), json_array("COMP")),"COMP")

2. Result

Re: Ingest time lookup to add fields does not work

PickleRick — Wed, 14 May 2025 05:52:02 GMT

@patelmc If the application field is a search-time extracted field it's unavailable during ingest-time processing. If you want to use it during indexing you have to first extract it as an indexed field (and can subsequently forget it so that it doesn't get indexed).

Bonus points question - why extracting those indexed fields?

@victor1004k Don't put settings in system/local. Put them into a separate app so they're easier to maintain.