topic Re: External lookup not working in Splunk Dev

External lookup not working

reswob10 — Wed, 30 Sep 2020 04:56:09 GMT

I've looked through several of the other posts on answers regarding this problem and I think I've tried all the suggestions, so here's my post:

I have a script I put $SPLUNK_HOME/etc/apps/search/bin as below:

splunk@splunk1:/opt/splunk/etc/apps/search/bin$ ll freq.py
-r-xr-xr-x 1 splunk splunk 657 Apr 10 20:33 freq.py*

It runs fine when testing with splunk python:

splunk@splunk1:/opt/splunk/etc/apps/search/bin$ /opt/splunk/bin/splunk cmd python ./freq.py splunk.com
domain,frequency
splunk.com,5.96996388594

I created a transforms.conf in $SPLUNK_HOME/etc/apps/search/local as below:

splunk@splunk1:/opt/splunk/etc/apps/search/local$ cat transforms.conf
[freqserver]
external_cmd = freq.py domain
external_type = external
fields_list = domain, frequency

Made sure it had the right linux permissions and owner:

splunk@splunk1:/opt/splunk/etc/apps/search/local$ ll
total 20
drwx------ 2 splunk splunk 4096 Apr 10 20:56 ./
drwxr-xr-x 10 splunk splunk 4096 Mar 10 21:03 ../
-rw------- 1 splunk splunk 807 Mar 30 00:49 indexes.conf
-rw------- 1 splunk splunk 122 Mar 10 21:49 inputs.conf
-rw------- 1 splunk splunk 101 Apr 10 20:56 transforms.conf

In the lookup definition, for permissions, it says that object should appear in all apps and everyone has read and write permissions.

I performed all the above as the admin of a single instance of Splunk.
I restarted Splunk.

So now I run a search:
index="bro" earliest=-1y sourcetype=bro_dns | fields query | rename query as domain | lookup freqserver domain

but I get the following error:

Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.

This is on splunk Version: 8.0.2

I was trying to follow these instructions for creating a new external lookup:
https[:]//docs.splunk.com/Documentation/Splunk/latest/Knowledge/Configureexternallookups

That error is the same error I get if I try a lookup name that does not exist:

index="bro" earliest=-1y sourcetype=bro_dns | fields query | rename query as domain | lookup nonsensename domain

would get the same kind of Could not construct lookup error...

Any suggestions?

Re: External lookup not working

to4kawa — Fri, 10 Apr 2020 21:40:24 GMT

How's search log?

Re: External lookup not working

reswob10 — Sat, 11 Apr 2020 14:56:24 GMT

here is the errors in the error log and some surrounding context..

04-11-2020 14:49:49.298 INFO  UnifiedSearch - Expanded index search = (index="bro" sourcetype=bro_dns _time>=1554994189.000)
04-11-2020 14:49:49.298 INFO  UnifiedSearch - base lispy: [ AND index::bro sourcetype::bro_dns ]
04-11-2020 14:49:49.298 INFO  UnifiedSearch - Processed search targeting arguments
04-11-2020 14:49:49.298 ERROR ExternalProvider - Command type 'external' is unsupported for lookup 'freqserver'.
04-11-2020 14:49:49.298 ERROR ExternalProvider - Command type 'external' is unsupported for lookup 'freqserver'.
04-11-2020 14:49:49.298 ERROR LookupProcessor - Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.
04-11-2020 14:49:49.298 ERROR SearchPhaseGenerator - Fallback to two phase search failed:Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.
04-11-2020 14:49:49.299 ERROR SearchOrchestrator - Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.
04-11-2020 14:49:49.299 ERROR SearchStatusEnforcer - sid:1586616589.121 Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.
04-11-2020 14:49:49.299 INFO  SearchStatusEnforcer - State changed to FAILED due to: Error in 'lookup' command: Could not construct lookup 'freqserver, domain'. See search.log for more details.
04-11-2020 14:49:49.299 INFO  SearchStatusEnforcer - Enforcing disk quota = 10485760000

Re: External lookup not working

to4kawa — Sat, 11 Apr 2020 20:44:46 GMT

see: transforms.conf

external_type = [python|executable|kvstore|geo|geo_hex]
* This setting describes the external lookup type.
* Use 'python' for external lookups that use a python script.
* Use 'executable' for external lookups that use a binary executable, such as a
  C++ executable.
* Use 'kvstore' for KV store lookups.
* Use 'geo' for geospatial lookups.
* 'geo_hex' is reserved for the geo_hex H3 lookup.
* Default: python

Re: External lookup not working

reswob10 — Wed, 30 Sep 2020 04:56:19 GMT

Thanks. This was the answer.

New transforms.conf

[freqserver]
external_cmd = freq.py domain
external_type = python
python.version = python2
fields_list = domain, frequency

Of course, if the script is updated to python3, change the setting accordingly.

Re: External lookup not working

to4kawa — Sun, 12 Apr 2020 01:53:06 GMT

thanks @reswob10

I was looking for a place to write and usage "python.version" .
In .conf , I'll write it.