Struggling with ingest-time lookup

PickleRick — Fri, 09 Apr 2021 08:24:44 GMT

Trying to do a lookup in ingest-time according to https://docs.splunk.com/Documentation/Splunk/8.1.3/Data/IngestLookups and can't get it to work.

If I do a simple transform like

[my-test-eval] INGEST_EVAL = test=spath(_raw,"Event.System.Computer")

I properly get the test field propagated with a value extracted from the event. (as you probably guessed - it's a typical windows XML-formed event).

But if I want to use the value retrieved from the event to perform a lookup... sorry, won't happen. And I have no clue why.

[my-test-eval] INGEST_EVAL = test=coalesce(json_extract(lookup("test.csv",json_object("Key",spath(_raw,"Event.System.Computer"),json_array("Value")),"Value"),"default")

There is no test field in the ingested event. Not even having the "default" value.

I tried giving the lookup name as defined in transforms.conf as well as the csv filename itself. I tried putting the lookup in app context as well as in system/local. Nothing works.

To make things more interesting - if I did some mistake in defining the lookup (like giving wrong column name), I'd get an error in the splunkd.log so it would be obvious that something is not right.

But the problem is I don't get any errors, the transform therefore should be working but it isn't. So I'm completely stuck here.

How to debug this thing?

Re: Struggling with ingest-time lookup

PickleRick — Fri, 09 Apr 2021 08:36:16 GMT

OK. I re-checked it after writing the post (it usually works this way - I struggle with something for a few hours, then ask on community, then have an epiphany 🤣). And it seems I had parentheses wrong in the eval expression. Which only shows how annoying this whole construct is because you don't get any errors in logs so it's hard to troubleshoot it.

topic Struggling with ingest-time lookup in Getting Data In

Struggling with ingest-time lookup

Re: Struggling with ingest-time lookup