Getting Data In

Ingest-time lookup

tah7004
Path Finder

Hello, has anyone worked with ingest-time lookup and familiar with it?

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestLookups

I'm confused on where the lookup is supposed to be.  Since this is an ingest-time process, I would think it would need to be in the indexers, but the doc isn't too clear on it.

Also regarding the actual stanza syntax, I'm trying to see if this works:

Lookup command:

lookup test field1 AS new_field1 field2 OUTPUT field3

[lookup-extract]
INGEST_EVAL= field3=json_extract(lookup("test", json_object("field1", new_field, "field2", field2), json_array("field3")),"field3")

Any help would be appreciated.

Labels (1)
0 Karma

tah7004
Path Finder

Some updates.  

I was able to get the lookup() function working via test searches.  My original lookup didn't work because it was too big at 1.5 G and I had to increase the max_mem_bytes in limits.conf.

Now, for the actual ingest-time lookup, I'm still not able to get it working with a test lookup file I created.  I think my initial struggles were due to some of the fields used for lookup are not indexed fields.  

I converted those fields as indexed fields using ingest_eval and also increased the ingest_max_mem_bytes as suggested by the doc.

Is there specific internal logs to watch out for as to why the ingest-time lookup failed?

I'm not having any luck digging through the _internal logs.

0 Karma

jpathak_splunk
Splunk Employee
Splunk Employee

You should be able to see relevant messages in splunkd.log which should be visible in _internal. As you pointed out, ingest time lookups depend on fields being present when events are retrieved from the index, are you sure those fields are index time fields ?

0 Karma

tah7004
Path Finder

Does the lookup have to be in $SPLUNK_HOME/etc/system/lookups?

I tried putting the lookup file and the props/transforms. conf in the indexers as an app, but that didn't work for me.

I also tried the lookup() function as an eval in test searches, but that isn't working.  I was following the lookup function guide here:

https://docs.splunk.com/Documentation/Splunk/8.1.1/SearchReference/ConditionalFunctions

 

0 Karma

The_Simko
SplunkTrust
SplunkTrust

Ingest-time lookups have to be on whatever server is first performing the parsing phase.  Normally that will be your indexer, but could also be on a heavy forwarder (or other Splunk Enterprise if they are where the data is being ingested).  
  
The Indexer (or other) will use their own knowledge objects, so get the lookup, props, and transforms on the server doing parsing.  

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...