I've followed http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups and looked at plenty of questions about the same topic on here and I still can't figure out what I'm doing wrong with my automatic lookup. I also watched a video on this but it didn't really show how the lookup was created.
Here's my csv file I want to use for a file based lookup:
gnovak@booberry:cat WAT_Lookups.csv "filename,description" "Invoice.pdf,Billing Invoice" "Statement.pdf,Billing Statement" "text.txt,Billing text" "*-*.pdf,Scorecard"
For Automatic Lookups, I have the following
Lookup Table: WAT_Lookups
Lookup input fields - filename = filename
Lookup Output fields - description = description
Apply to : sourcetype named EPPWEB
I have checked my props.conf and transforms.conf files after configuring all of this and there are entires in there. I also made sure the permissions on these were all Everyone can Read, Admin can write for only the search app which is where this is located.
When I do a search for sourcetype=EPPWEB, I get the following error:
[log1.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups' [log2.blahblahblah.info] Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'source::EPPWEB' and lookup table 'WAT_Lookups'
I just can't seem to get it to work.
Basically the end result is, for example, a filename called Invoice.pdf to be otherwise known as "Billing Invoice".
NOTE: I already have "filename" as a field extracted through props.conf.
So under the field filename you have some files listed like text.text, Invoice.pdf, etc. I'm not sure if this in doing anything w/ the lookup.
The solution to this problem was that the original search did not have enough information in it to do the lookup. The search that allowed the "description" field to show up was:
sourcetype=EPPWEB source="/opt/log/*/web_server/info.log" WAT | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description
It just needed more information I guess. The automatic lookup works now. Thanks for your input and assistance though! I learned a lot. 🙂
If you can see filename show up then it's not a problem. I would suggest recreating the steps to create the lookup and delete the old ones. Do it as a manual and try it from the search and then make it automatic.
I even tried this search and it didn't work:
sourcetype=EPPWEB | lookup WAT_Lookups filename AS filename OUTPUTNEW description AS description
It should look at a name in the "filename" field and match it up wtih the name in the description field (based on what's in the csv file). I don't see a description field at all.
It searches and brings back results but there is no "description" field with the names i specified. And the lookup definition was called WAT_Lookups. I'm not sure if "where" my field extraction is located is the problem? My field extraction for "filename" is located in /opt/splunk/etc/system/local. This lookup is in /opt/splunk/etc/apps/search/local.
Here's the extraction in props.conf for "filename"
EXTRACT-extract_my_fields = USER (?P
SHOULD_LINEMERGE = FALSE
I assume 'filename' is a field that exists for your sourcetype. Does the description field appear if you do this search? Assuming that WAT_Lookups is the name of the look up in Manager » Lookups » Lookup definitions.
sourcetype='EPPWEB' | lookup WAT_Lookups filename
If this works then there is something wrong with your automatic look up. Just seems to be a configuration issue here somewhere. Splunk shouldn't do anything to the file so it must have gotten put in there by your editor.