Splunk Search

Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff
Path Finder

Searches with lookups are failing in our environment. I have created a lookup file called dt1.csv and a lookup definition called dt1. Both the file and the definition have read and write permissions for all users in the search app.

This works successfully and shows the contents of the lookup:

|inputlookup dt1 

However, using the lookup as part of a query such as:

<search> |lookup dt1 cs_host

Fails with these errors:

[indexer1] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.
[indexer2] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.

Every indexer errors out with that message. This happens regardless of the app the lookup is created in, and regardless of the user that creates the lookup.

Any ideas on what would cause lookups to fail with these errors? We're on Splunk 6.5.1.

0 Karma
1 Solution

dewoodruff
Path Finder

The root cause ended up being a custom app with very long file paths caused the knowledge bundle to fail to replicate. Once the app was removed, the knowledge bundle began replicating again and lookups started working.

The is apparently a known issue with 6.5.1 and has been resolved in some later version.

View solution in original post

dewoodruff
Path Finder

The root cause ended up being a custom app with very long file paths caused the knowledge bundle to fail to replicate. Once the app was removed, the knowledge bundle began replicating again and lookups started working.

The is apparently a known issue with 6.5.1 and has been resolved in some later version.

aaraneta_splunk
Splunk Employee
Splunk Employee

@dewoodruff - Glad you found the solution to your question. Please don't forget to click "Accept" to close out your question and upvote any answers/comments that were helpful. Thanks!

0 Karma

evertonpsp
New Member

Helo, Can You help me?

I have a like problem, but i think that the root cause is permission, since the problem happens for the users of a certain Role and the others work.

The file (Lookup table files) is read-only for all users and apps.

The Role Capabilities of the user that works is different from the Role Capabilities of the problem user, but I do not know which one I should add so that both work.

Does anyone have any ideas?
The problem can be Capabilities?

Thanks!

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Okay, so IIRC, "local" forces the lookup action to be executed on the search head, while the other does not. Doesn't that indicate that the lookup table is not being replicated to the indexers/peers?

https://answers.splunk.com/answers/343835/how-to-distribute-lookup-tables-in-an-indexer-clus.html
https://answers.splunk.com/answers/634/in-a-distributed-search-environment-where-do-my-configuration...

This thread describes a similar issue when the lookup table name was too long to be bundled. (obviously not the case here.)
https://answers.splunk.com/answers/200719/where-does-a-lookup-table-need-to-be-in-a-distribu.html

In any case, on several threads I saw the admonition, "Check your bundle replication error messages." Something in the replication process is screwy.

Lastly, the lesson learned in this one was that the new/altered lookup table had to be on the SH captain, not just any search head.
https://answers.splunk.com/answers/338008/why-do-i-see-old-data-in-my-lookup-table-in-a-sear.html

adonio
Ultra Champion

try and do your search and | lookup dt1.csv cs_host

0 Karma

woodcock
Esteemed Legend

See the answer by @sjohnson. One way to test is to try this which will work but it will be slower than it should be:

 <search> |lookup local=true dt1 cs_host

dewoodruff
Path Finder

We're not able to find a distsearch.conf file in the search app directory. The query does succeed using local=true.

0 Karma

woodcock
Esteemed Legend

Then you have a permission or (app) scope problem and you must not be running the 2 searches as the same user in the same app. Show me the URL for both |inputlookup dt1, <search> |lookup dt1 cs_host, and <search> |lookup local=true dt1 cs_host and make sure that you are logged in as the same user each time. It is surely that you are in 2 different apps; one which has access to the lookup and the other which does not.

0 Karma

adauria_splunk
Splunk Employee
Splunk Employee

They were run both as the same regular user, and as the same administrative user, with the same results. Everything was done within the search app only.

0 Karma

dewoodruff
Path Finder

Here you are. Hostname and index name removed for privacy.

inputlookup:

<hostname>/en-US/app/search/search?q=%7Cinputlookup%20dt1&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=statistics&sid=1494609014.208099&display.page.search.tab=statistics

index= |lookup dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609057.208159

index= |lookup local=true dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20local%3Dtrue%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609075.208172
0 Karma

woodcock
Esteemed Legend

What version of splunk?

0 Karma

woodcock
Esteemed Legend

Are you clustered?

0 Karma

dewoodruff
Path Finder

6.5.1. Search head is standalone. There are multiple indexers.

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Is there a distsearch.conf in the search app that has a blacklist for lookups?

adauria_splunk
Splunk Employee
Splunk Employee

I am the SE assisting here. No, we can't seem to find any distsearch.conf outside of the default directories. I don't see anything in there blacklisting these lookups.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...