ingest_eval lookup example

[schose]

Hi all,

I'm trying to ingest data using a lookup like descripted in: 

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestLookups

props.conf:

[ilookuptest]

TRANSFORMS-a = ilookuptest1

TRANSFORMS-b = ilookuptest2

transforms.conf:

[ilookuptest1]

INGEST_EVAL = pod="testpod1"

[ilookuptest2]

INGEST_EVAL= annotation=lookup("testlookup.csv", json_object("pod","pod"), json_array("annotation"))

lookup testlookup.csv:

pod,annotation
testpod1,testannotation1
testpod2,testannotation2

ingest data using:

curl -k http://192.168.208.5:8088/services/collector -H 'Authorization: Splunk f05eedbb-a706-427e-9606-baa3e8036411' -d '{"index": "test", "sourcetype": "ilookuptest", "event":"this is for testing ingest eval lookup12"}

props.conf and transforms.conf are located at $SPLUNK_HOME/etc/system/local .. lookup at $SPLUNK_HOME/etc/system/lookups . 

I'm getting errors  in splunkd.log:

WARN CsvDataProvider - No valid lookup table file found for this lookup=testlookup

ERROR CsvDataProvider - The lookup table 'testlookup' does not exist or is not available.

ERROR pipeline - Runtime exception in pipeline=typing processor=regexreplacement error='Invalid function argument' confkey='source::http:test|host::192.168.208.5:8088|ilookuptest|'
ERROR pipeline - Uncaught exception in pipeline execution (regexreplacement) - getting next event

The event is not indexed...

When defining transforms.conf
INGEST_EVAL= annotation=lookup("testlookup", json_object("pod","pod"), json_array("annotation"))

I'm getting errors in splunkd.log:

WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename.

Event is indexed but not getting the value from the lookup. 

File is there, read permissions are set, "| inputlookup testlookup.csv" is displaying results.

Any hints or a working INGEST_EVAL using lookups example?

Best Regards,

Andreas

[schose · Accepted solution]

a working transforms.conf:

[ilookuptest1]

INGEST_EVAL = pod="testpod1"

[ilookuptest2]

INGEST_EVAL= annotation=json_extract(lookup("testlookup.csv",json_object("pod",pod), json_array(annotation)),"annotation")

message:

WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename.

still there, but working.

Regards,

Andreas

[tah7004]

Hi, do you know what made it work for you? I get the same WARN message, but not the error and I think my configuration is similar. I tried placing the lookup file in both the app and system/lookup directory on my indexers.