Error 'Could not find all of the specified lookup fields in the lookup table.'

[danielpellarini]

I have created a lookup table to substitute some values in Splunk with some new values in the lookup table, but when I run a search that should make use of this lookup table I get the error message Error 'Could not find all of the specified lookup fields in the lookup table.'.

So then I tried creating a dummy lookup table with dummy values to see if that worked, but that didn't work either. So this is everything I have done, I'm hoping it might help you helping me 🙂

First things first: I have already tried the suggestions in this question and in this question, but none of them worked (i.e. adding a third column and getting rid of hidden characters).

The dummy lookup table is called TaskCategory.csv and I have put it into /opt/splunk/etc/apps/search/lookups. Its contents are simply

task_category,task_category_new_value
Logon,Lookup_Value_1

Inside /opt/splunk/etc/apps/search/local I have created transforms.conf and props.conf. This is transforms.conf:

[TaskCategory]
filename = TaskCategory.csv

And this is props.conf:

[WinEventLog:Security]
LOOKUP-AutoTaskCategory = TaskCategory TaskCategory AS task_category OUTPUT task_category_new_value

Now, after restarting Splunk, if I run a search like sourcetype=WinEventLog:Security | top TaskCategory I get the error message. Running | inputlookup TaskCategory works without errors (i.e. I see the lookup table correctly displayed.

This is all there is in my config files and all the information should make it easy for you to recreate this scenario.

Any suggestions on how to solve this?

[sowings · Accepted solution]

Reverse the order of terms in the AS phrase of your props.conf line. That is, the first portion of the lookup definition should read like "this_field_is_in_the_lookup AS this_field_is_in_the_data".

[nwieseler]

the first portion of the lookup definition should read like "this_field_is_in_the_lookup AS this_field_is_in_the_data".

Thanks for the snip above!

Nick

[sowings]

Hmm, using OUTPUT (vs. OUTPUTNEW) should be overwriting the TaskCategory, but maybe because it was used as the lookup key field itself... I don't know offhand. You could try reversing the order here on the OUTPUT side as well, or my approach would be to simply rename the field in the lookup.

More info can be found here:

http://docs.splunk.com/Documentation/Splunk/6.0/SearchReference/Lookup

[danielpellarini]

Brilliant, thank you! What if I want to overwrite the value of TaskCategory with task_category_new_value? Doing OUTPUT task_category_new_value AS TaskCategory doesn't work, is there a way to do this? Upvote for now.

[papemalik]

Hello. my fields name are the same, but still it's not working.
my working with access log and a malware domain list.
Thanks