Splunk Search

Automatic Lookup not working

Builder

I've followed http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups and looked at plenty of questions about the same topic on here and I still can't figure out what I'm doing wrong with my automatic lookup. I also watched a video on this but it didn't really show how the lookup was created.

Here's my csv file I want to use for a file based lookup:

gnovak@booberry:cat WAT_Lookups.csv

"filename,description"
"Invoice.pdf,Billing Invoice"
"Statement.pdf,Billing Statement"
"text.txt,Billing text"
"*-*.pdf,Scorecard"
  1. For Lookup Table Files I selected this csv and gave it the same name for Destination filename.
  2. For Lookup Definitions, destination app is "search", name is "WAT_Lookups.csv", type is "file based", and the lookup file is "WAT_Lookups.csv".
  3. For Automatic Lookups, I have the following

    Lookup Table: WAT_Lookups
    Lookup input fields - filename = filename
    Lookup Output fields - description = description
    Apply to : sourcetype named EPPWEB

I have checked my props.conf and transforms.conf files after configuring all of this and there are entires in there. I also made sure the permissions on these were all Everyone can Read, Admin can write for only the search app which is where this is located.

When I do a search for sourcetype=EPPWEB, I get the following error:

[log1.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups'
    [log2.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups'

I just can't seem to get it to work.

Basically the end result is, for example, a filename called Invoice.pdf to be otherwise known as "Billing Invoice".

NOTE: I already have "filename" as a field extracted through props.conf.

So under the field filename you have some files listed like text.text, Invoice.pdf, etc. I'm not sure if this in doing anything w/ the lookup.

Tags (2)

Splunk Employee
Splunk Employee

Can you try it without the double quotes in your look up file. I'm guessing that is causing issues.

filename,description
Invoice.pdf,Billing Invoice
Statement.pdf,Billing Statement

0 Karma

Builder

yeah...see below.

0 Karma

Builder

The solution to this problem was that the original search did not have enough information in it to do the lookup. The search that allowed the "description" field to show up was:

sourcetype=EPPWEB source="/opt/log/*/web_server/info.log" WAT | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description

It just needed more information I guess. The automatic lookup works now. Thanks for your input and assistance though! I learned a lot. 🙂

0 Karma

Splunk Employee
Splunk Employee

Did you get it?

0 Karma

Builder

I actually did try recreating the automatic lookup and i got the same result. I could try manual i guess

0 Karma

Splunk Employee
Splunk Employee

If you can see filename show up then it's not a problem. I would suggest recreating the steps to create the lookup and delete the old ones. Do it as a manual and try it from the search and then make it automatic.

0 Karma

Builder

I even tried this search and it didn't work:
sourcetype=EPPWEB | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description

It should look at a name in the "filename" field and match it up wtih the name in the description field (based on what's in the csv file). I don't see a description field at all.

0 Karma

Builder

It searches and brings back results but there is no "description" field with the names i specified. And the lookup definition was called WAT_Lookups. I'm not sure if "where" my field extraction is located is the problem? My field extraction for "filename" is located in /opt/splunk/etc/system/local. This lookup is in /opt/splunk/etc/apps/search/local.

Here's the extraction in props.conf for "filename"

[EPPWEB]
EXTRACT-extract_my_fields = USER (?P[\d+-\w\w]) downloading .*\/(?.+?)$
SHOULD_LINEMERGE = FALSE

0 Karma

Splunk Employee
Splunk Employee

I assume 'filename' is a field that exists for your sourcetype. Does the description field appear if you do this search? Assuming that WAT_Lookups is the name of the look up in Manager » Lookups » Lookup definitions.

sourcetype='EPPWEB' | lookup WAT_Lookups filename

If this works then there is something wrong with your automatic look up. Just seems to be a configuration issue here somewhere. Splunk shouldn't do anything to the file so it must have gotten put in there by your editor.

0 Karma

Builder

And not sure why splunk put the "" in the file. The original one they were not in there.

0 Karma

Builder

The error is gone now but there still isn't a description field.

0 Karma