topic Re: My lookup in a macro works in a search, but why does it not work using savedsearch command? in Splunk Search

My lookup in a macro works in a search, but why does it not work using savedsearch command?

rgsage — Mon, 06 Feb 2017 18:17:08 GMT

We are on Splunk 6.2.1. This is all in Splunk search...

I have a macro with lookup which works fine in a simple search but when I save the search and attempt to invoke from | savedsearch ... I get this error:

Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table.

Details: lookup file my_test.csv like this:

key,value
A,1
B,2

And very simple macro my_macro like this:

lookup my_test.csv key OUTPUT value

And very simple search "my_search" like this:

index=my_index [edited]
| eval key = "A"
| `my_macro` 
| table _time key value

So far everything works fine. All rows have key=A, value=1.

When I do:

| savedsearch "my_search"

I get error message:

"Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table."

Job inspector shows eventSearch (only):

search index=my_index [Edited] | eval key = "A"

Any ideas?

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

s2_splunk — Mon, 06 Feb 2017 18:27:20 GMT

Permissions...? Is the saved search running in an app context that has visibility to the macro? I suspect the macro is in your private user space and not visible by the saved search. Try setting permissions for your macro to 'global' and see if that changes things.

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

rgsage — Mon, 06 Feb 2017 18:33:06 GMT

Thank you for quick reply. Sorry I should have included in the details my_macro is saved as Sharing=Global, with Permissions: Read access for Everyone.

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

s2_splunk — Mon, 06 Feb 2017 18:54:16 GMT

Hmm, I just tested this on my local 6.5.0 instance without issues.
Take a look at the search.log for your execution (there's a link to it a the top (and bottom) of the Job Inspector window and see if you can find any hints.

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

somesoni2 — Mon, 06 Feb 2017 19:03:17 GMT

What about lookup table file permissions?

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

rgsage — Mon, 06 Feb 2017 19:16:02 GMT

The only interesting lines from search.log are:
...
02-06-2017 11:12:35.917 INFO LookupOperator - Assuming table name 'my_test.csv' is implicit filename.
02-06-2017 11:12:35.918 ERROR LookupOperator - Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table.
...
02-06-2017 11:12:35.930 INFO ShutdownHandler - Shutdown complete in 281 microseconds
02-06-2017 11:12:35.931 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table.
[End]

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

rgsage — Mon, 06 Feb 2017 19:18:19 GMT

Lookup file was added directly to the filesystem (as with all our lookups), and looks like (from Settings/Lookups/Lookup table files page) in that case Splunk defaults to Sharing=Global.

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

DalJeanis — Mon, 06 Feb 2017 20:32:24 GMT

In general, programmers should avoid using keywords as variable names. You never know when such mixed nomenclature will confuse the data plumbing. I'm suspecting that the issue in this case is either the variable name "value" or the action of the OUTPUT key word.

Try changing the code to the following -

inkey,outvalue
 A,1
 B,2


 lookup my_test.csv inkey as mykey OUTPUT outvalue as myvalue


index=my_index [edited]
 | eval mykey = "A"
 | eval myvalue = "whocares"
 | `my_macro` 
 | table _time mykey myvalue

If either of my suspected issues is correct, that code should fix it.

(The variable names I wanted changed because I would normally use fillnull to set the value for the field "value", and value is a keyword used to set the value you are going to fill for nulls.)

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

rgsage — Tue, 07 Feb 2017 01:54:07 GMT

Thank you for your reply! Turns out your second guess was correct: OUTPUT was the culprit. For some reason when used with savedsearch, the lookup must have an explicit OUTPUT foo AS something.

In my case this macro works in normal search, but not from savedsearch:

lookup my_test.csv key OUTPUT value

This macro works in normal search, AND from savedsearch:

lookup my_test.csv key OUTPUT value as value

Here is an interesting additional wrinkle that may help someone... If the lookup macro has a trailing space ("lookup my_test.csv OUTPUT value AS value ") it will again work in normal search but again NOT from savedsearch. Wow.

Perhaps @ssievert did not see this behavior because there are some Splunk bugs here that have been fixed since 6.2.1... Thanks again!

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

DalJeanis — Tue, 07 Feb 2017 18:47:36 GMT

You're welcome. Yes, I've come to trust that, if reading something makes me ITCH, there's probably a hidden gotcha.

Twenty years ago, I'd tell my junior programmers when they showed me a creative way of doing something, "Well, I wouldn't do it that way, but it ought to work."

Often, they'd be back an hour later to tell me about their new problem, and I'd remember. "Oh, yeah, THAT'S why I wouldn't do it that way."

Nowadays, I just raise my eyebrow and say, "THIS PART makes me itch. I'd rewrite that to do X, Y and Z." Usually, they do.

Re: My lookup in a macro works in a search, but why does it not work using savedsearch command?

DalJeanis — Tue, 07 Feb 2017 18:49:47 GMT

Of course, if somesoni2 shows me a creative way to do something, then I study it until it feels natural, find another application for it, then I put it in my toolbox.