Getting Data In

ingest_eval lookup example

schose
Builder

Hi all,

I'm trying to ingest data using a lookup like descripted in: 

https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestLookups

props.conf:

[ilookuptest]
TRANSFORMS-a = ilookuptest1
TRANSFORMS-b = ilookuptest2
 
transforms.conf:
 
[ilookuptest1]
INGEST_EVAL = pod="testpod1"

[ilookuptest2]
INGEST_EVAL= annotation=lookup("testlookup.csv", json_object("pod","pod"), json_array("annotation"))
 
lookup testlookup.csv:
 
pod,annotation
testpod1,testannotation1
testpod2,testannotation2
 
ingest data using:
curl -k http://192.168.208.5:8088/services/collector -H 'Authorization: Splunk f05eedbb-a706-427e-9606-baa3e8036411' -d '{"index": "test", "sourcetype": "ilookuptest", "event":"this is for testing ingest eval lookup12"}
 
props.conf and transforms.conf are located at $SPLUNK_HOME/etc/system/local .. lookup at $SPLUNK_HOME/etc/system/lookups . 
 
I'm getting errors  in splunkd.log:
WARN CsvDataProvider - No valid lookup table file found for this lookup=testlookup
ERROR CsvDataProvider - The lookup table 'testlookup' does not exist or is not available.
ERROR pipeline - Runtime exception in pipeline=typing processor=regexreplacement error='Invalid function argument' confkey='source::http:test|host::192.168.208.5:8088|ilookuptest|'
ERROR pipeline - Uncaught exception in pipeline execution (regexreplacement) - getting next event

The event is not indexed...
 
When defining transforms.conf
INGEST_EVAL= annotation=lookup("testlookup", json_object("pod","pod"), json_array("annotation"))
 
I'm getting errors in splunkd.log:
WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename.
 
Event is indexed but not getting the value from the lookup. 
 
File is there, read permissions are set, "| inputlookup testlookup.csv" is displaying results.
 
Any hints or a working INGEST_EVAL using lookups example?
 
Best Regards,
 
Andreas
Labels (2)
0 Karma
1 Solution

schose
Builder

a working transforms.conf:

[ilookuptest1]

INGEST_EVAL = pod="testpod1"

[ilookuptest2]
INGEST_EVAL= annotation=json_extract(lookup("testlookup.csv",json_object("pod",pod), json_array(annotation)),"annotation")
 
message:
WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename.

still there, but working.
 
Regards,
 
Andreas

View solution in original post

Tags (1)
0 Karma

schose
Builder

a working transforms.conf:

[ilookuptest1]

INGEST_EVAL = pod="testpod1"

[ilookuptest2]
INGEST_EVAL= annotation=json_extract(lookup("testlookup.csv",json_object("pod",pod), json_array(annotation)),"annotation")
 
message:
WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename.

still there, but working.
 
Regards,
 
Andreas

View solution in original post

Tags (1)
0 Karma

tah7004
Path Finder
Hi, do you know what made it work for you? I get the same WARN message, but not the error and I think my configuration is similar. I tried placing the lookup file in both the app and system/lookup directory on my indexers.
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!