topic Re: Ingest-time lookup in Getting Data In

Ingest-time lookup

tah7004 — Fri, 29 Jan 2021 20:45:27 GMT

Hello, has anyone worked with ingest-time lookup and familiar with it?

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestLookups

I'm confused on where the lookup is supposed to be. Since this is an ingest-time process, I would think it would need to be in the indexers, but the doc isn't too clear on it.

Also regarding the actual stanza syntax, I'm trying to see if this works:

Lookup command:

lookup test field1 AS new_field1 field2 OUTPUT field3

[lookup-extract]
INGEST_EVAL= field3=json_extract(lookup("test", json_object("field1", new_field, "field2", field2), json_array("field3")),"field3")

Any help would be appreciated.

Re: Ingest-time lookup

The_Simko — Sun, 31 Jan 2021 04:04:33 GMT

Ingest-time lookups have to be on whatever server is first performing the parsing phase. Normally that will be your indexer, but could also be on a heavy forwarder (or other Splunk Enterprise if they are where the data is being ingested).

The Indexer (or other) will use their own knowledge objects, so get the lookup, props, and transforms on the server doing parsing.

Re: Ingest-time lookup

tah7004 — Mon, 01 Feb 2021 14:11:23 GMT

Does the lookup have to be in $SPLUNK_HOME/etc/system/lookups?

I tried putting the lookup file and the props/transforms. conf in the indexers as an app, but that didn't work for me.

I also tried the lookup() function as an eval in test searches, but that isn't working. I was following the lookup function guide here:

https://docs.splunk.com/Documentation/Splunk/8.1.1/SearchReference/ConditionalFunctions

Re: Ingest-time lookup

tah7004 — Tue, 02 Feb 2021 14:05:59 GMT

Some updates.

I was able to get the lookup() function working via test searches. My original lookup didn't work because it was too big at 1.5 G and I had to increase the max_mem_bytes in limits.conf.

Now, for the actual ingest-time lookup, I'm still not able to get it working with a test lookup file I created. I think my initial struggles were due to some of the fields used for lookup are not indexed fields.

I converted those fields as indexed fields using ingest_eval and also increased the ingest_max_mem_bytes as suggested by the doc.

Is there specific internal logs to watch out for as to why the ingest-time lookup failed?

I'm not having any luck digging through the _internal logs.

Re: Ingest-time lookup

jpathak_splunk — Fri, 13 Aug 2021 18:47:57 GMT

You should be able to see relevant messages in splunkd.log which should be visible in _internal. As you pointed out, ingest time lookups depend on fields being present when events are retrieved from the index, are you sure those fields are index time fields ?

Re: Ingest-time lookup

victor1004k — Mon, 26 May 2025 09:04:39 GMT

@tah7004 To use ingest-time lookup, the field you want to apply must be specified as an indexed-field. You can apply it successfully by configuring the configuration file as follows.

1. $SPLUNK_HOME/etc/apps/myapp/lookups/test.csv

field1,field2,field3 value1,value2,value3

2. $SPLUNK_HOME/etc/apps/myapp/local/props.conf

[test_ingest_lookup] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = true TRANSFORMS-ingest_time_lookup = regex_extract_av_pairs, lookup_extract

3. $SPLUNK_HOME/etc/apps/myapp/local/transforms.conf

[regex_extract_av_pairs] SOURCE_KEY = _raw REGEX = \s([a-zA-Z][a-zA-Z0-9-]+)=([^\s"',]+) REPEAT_MATCH = true FORMAT = $1::"$2" WRITE_META = true [lookup_extract] INGEST_EVAL= field3=json_extract(lookup("test.csv", json_object("field1", new_field, "field2", field2), json_array("field3")),"field3")

You can refer to another solution using INDEXED_EXTRACTIONS=json in the link below.

- Splunkデータ取り込み時の絞り込み方法(リストマッチ)
https://qiita.com/chobiyu/items/aec5ef3a75a8bab96546

Re: Ingest-time lookup

PickleRick — Tue, 13 May 2025 07:24:11 GMT

Not necessarily. You can use an output of a function operating on _raw as argument to the lookup() function.

Re: Ingest-time lookup

victor1004k — Mon, 26 May 2025 09:04:59 GMT

@tah7004 OK! Bellow is the answer you talk about.

1. $SPLUNK_HOME/etc/apps/myapp/local/props.conf

TRANSFORMS-ingest_time_lookup = lookup_extract

2. $SPLUNK_HOME/etc/apps/myapp/local/transforms.conf

[lookup_extract] INGEST_EVAL= field1=replace(_raw, ".*field1=([0-9A-Za-Z.]+).*", "\1"), field2=replace(_raw, ".*field2=([0-9A-Za-Z.]+).*", "\1"), field3=json_extract(lookup("test.csv", json_object("field1", new_field, "field2", field2), json_array("field3")),"field3")

Re: Ingest-time lookup

isoutamo — Fri, 16 May 2025 15:58:58 GMT

One another comment. Don't use .../etc/system/local for (almost) anything! Create your own app and use it to store your conf files. In that way everything is working much better in long run.