Splunk Search

external_lookup example: Could not find all of the specified lookup fields in the lookup table

are0002
Path Finder

I got this error when I configure an automathic lookup:
Could not find all of the specified lookup fields in the lookup table

I tried the script and runs perfectly
I created a partial CSV file and execute it with the python srcript:
/opt/splunk/bin/splunk cmd python /opt/splunk/etc/system/bin/external_lookup.py host ip < temp.csv
host,ip
www.hardware.fr,83.243.20.80
www.bash.org,69.61.106.93
www.somafm.com,64.147.167.20

Then I configured the trasnforms.conf:

and the props.conf:
[mydnslookup]
external_cmd = external_lookup.py host ip
external_type = python
fields_list = dst_ip, dst_hostname

[pan_traffic]
LOOKUP-rdns = mydnslookup ip AS dst_ip OUTPUT host AS resolved_host

also I tried:

[pan_traffic]
LOOKUP-rdns = mydnslookup clientip AS dst_ip OUTPUT host AS resolved_host

I used this tutorial:
http://docs.splunk.com/Documentation/Splunk/latest/knowledge/Addfieldsfromexternaldatasources#Set_up...

0 Karma
1 Solution

lguinn2
Legend

Try

fields_list = host, ip

instead of

fields_list = dst_ip, dst_hostname

And

[pan_traffic]
LOOKUP-rdns = mydnslookup ip AS dst_ip OUTPUT host AS resolved_host

is correct. I assume that the [pan_traffic] stanza is in props.conf and the [mydnslookup] is in transforms.conf

Finally, I assume that the dst_ip field exists in your pan_traffic sourcetype. It can't be a field that you have extracted with rex. AND, order is important: field definition for dst_ip should precede the lookup in props.conf

View solution in original post

are0002
Path Finder

Hi Iguinn,

You were right. The field_list refers to the CSV headers instead of splunk data fields. Thank you for your answer.

0 Karma

lguinn2
Legend

Try

fields_list = host, ip

instead of

fields_list = dst_ip, dst_hostname

And

[pan_traffic]
LOOKUP-rdns = mydnslookup ip AS dst_ip OUTPUT host AS resolved_host

is correct. I assume that the [pan_traffic] stanza is in props.conf and the [mydnslookup] is in transforms.conf

Finally, I assume that the dst_ip field exists in your pan_traffic sourcetype. It can't be a field that you have extracted with rex. AND, order is important: field definition for dst_ip should precede the lookup in props.conf

are0002
Path Finder

I am trying to do a reverse dns resolution (obtain the domain name with the IP address).
I am using the external_lookup.py script that is able to do dns resolutions in both ways. I need to resolve the dns dynamicaly that's why I need an external script.

0 Karma

lguinn2
Legend

First, I don't think I understand what you are trying to do.

If you want to have Splunk look up IP addresses in a fixed file, you don't need external_lookup.py

A file-based lookup is much easier than what you are doing. There is documentation for creating a file-based lookup at Setup a fields lookup based on a static file . The documentation shows how to edit props.conf and transforms.conf

But you can do this very easily from the Manager UI:

  1. Build a CSV file on your desktop that contains the data you need. The first row MUST be a header; the column names will be the field names in your lookup.
  2. Go to Manager » Lookups in the Splunk UI
  3. Add a new Lookup Table File. This is where you will upload the CSV file from your desktop into Splunk.
  4. Add a new Lookup Definition. This is where you tell Splunk that you want to do a file-based lookup, using your Lookup Table file from the previous step.
  5. Add a new Automatic Lookup. Here you tell Splunk how to use your Lookup Definition automatically, and tell it which fields to retrieve, etc.
  6. Be sure to set permissions on each of the items: the table, the definition, and the automatic lookup - if you want them to be used by others.

Once you create the lookup in the UI, you can see what it does in props.conf and transforms.conf - but no need to create them yourself.

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...