topic Re: Automatic Lookup not working in Splunk Search

Automatic Lookup not working

gnovak — Mon, 28 Sep 2020 12:24:45 GMT

I've followed http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups and looked at plenty of questions about the same topic on here and I still can't figure out what I'm doing wrong with my automatic lookup. I also watched a video on this but it didn't really show how the lookup was created.

Here's my csv file I want to use for a file based lookup:

gnovak@booberry:cat WAT_Lookups.csv

"filename,description"
"Invoice.pdf,Billing Invoice"
"Statement.pdf,Billing Statement"
"text.txt,Billing text"
"*-*.pdf,Scorecard"

For Lookup Table Files I selected this csv and gave it the same name for Destination filename.
For Lookup Definitions, destination app is "search", name is "WAT_Lookups.csv", type is "file based", and the lookup file is "WAT_Lookups.csv".
For Automatic Lookups, I have the following

Lookup Table: WAT_Lookups
Lookup input fields - filename = filename
Lookup Output fields - description = description
Apply to : sourcetype named EPPWEB

I have checked my props.conf and transforms.conf files after configuring all of this and there are entires in there. I also made sure the permissions on these were all Everyone can Read, Admin can write for only the search app which is where this is located.

When I do a search for sourcetype=EPPWEB, I get the following error:

[log1.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups'
    [log2.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups'

I just can't seem to get it to work.

Basically the end result is, for example, a filename called Invoice.pdf to be otherwise known as "Billing Invoice".

NOTE: I already have "filename" as a field extracted through props.conf.

So under the field filename you have some files listed like text.text, Invoice.pdf, etc. I'm not sure if this in doing anything w/ the lookup.

Re: Automatic Lookup not working

sdaniels — Mon, 10 Sep 2012 21:02:12 GMT

Can you try it without the double quotes in your look up file. I'm guessing that is causing issues.

filename,description
Invoice.pdf,Billing Invoice
Statement.pdf,Billing Statement

Re: Automatic Lookup not working

gnovak — Mon, 10 Sep 2012 21:10:48 GMT

The error is gone now but there still isn't a description field.

Re: Automatic Lookup not working

gnovak — Mon, 10 Sep 2012 21:12:27 GMT

And not sure why splunk put the "" in the file. The original one they were not in there.

Re: Automatic Lookup not working

sdaniels — Mon, 10 Sep 2012 21:17:34 GMT

I assume 'filename' is a field that exists for your sourcetype. Does the description field appear if you do this search? Assuming that WAT_Lookups is the name of the look up in Manager » Lookups » Lookup definitions.

sourcetype='EPPWEB' | lookup WAT_Lookups filename

If this works then there is something wrong with your automatic look up. Just seems to be a configuration issue here somewhere. Splunk shouldn't do anything to the file so it must have gotten put in there by your editor.

Re: Automatic Lookup not working

gnovak — Mon, 28 Sep 2020 12:24:47 GMT

It searches and brings back results but there is no "description" field with the names i specified. And the lookup definition was called WAT_Lookups. I'm not sure if "where" my field extraction is located is the problem? My field extraction for "filename" is located in /opt/splunk/etc/system/local. This lookup is in /opt/splunk/etc/apps/search/local.

Here's the extraction in props.conf for "filename"

[EPPWEB]
EXTRACT-extract_my_fields = USER (?P[\d+-\w\w]) downloading .*\/(?.+?)$
SHOULD_LINEMERGE = FALSE

Re: Automatic Lookup not working

gnovak — Mon, 10 Sep 2012 21:48:46 GMT

I even tried this search and it didn't work:
sourcetype=EPPWEB | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description

It should look at a name in the "filename" field and match it up wtih the name in the description field (based on what's in the csv file). I don't see a description field at all.

Re: Automatic Lookup not working

sdaniels — Mon, 10 Sep 2012 21:57:54 GMT

If you can see filename show up then it's not a problem. I would suggest recreating the steps to create the lookup and delete the old ones. Do it as a manual and try it from the search and then make it automatic.

Re: Automatic Lookup not working

gnovak — Mon, 10 Sep 2012 22:00:24 GMT

I actually did try recreating the automatic lookup and i got the same result. I could try manual i guess

Re: Automatic Lookup not working

sdaniels — Tue, 11 Sep 2012 14:35:44 GMT

Did you get it?

Re: Automatic Lookup not working

gnovak — Mon, 28 Sep 2020 12:25:11 GMT

The solution to this problem was that the original search did not have enough information in it to do the lookup. The search that allowed the "description" field to show up was:

sourcetype=EPPWEB source="/opt/log/*/web_server/info.log" WAT | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description

It just needed more information I guess. The automatic lookup works now. Thanks for your input and assistance though! I learned a lot. 🙂

Re: Automatic Lookup not working

gnovak — Tue, 11 Sep 2012 18:42:43 GMT

yeah...see below.