topic Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error? in Splunk Search

Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff — Wed, 10 May 2017 18:20:38 GMT

Searches with lookups are failing in our environment. I have created a lookup file called dt1.csv and a lookup definition called dt1. Both the file and the definition have read and write permissions for all users in the search app.

This works successfully and shows the contents of the lookup:

|inputlookup dt1

However, using the lookup as part of a query such as:

<search> |lookup dt1 cs_host

Fails with these errors:

[indexer1] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.
[indexer2] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'dt1' does not exist or is not available.

Every indexer errors out with that message. This happens regardless of the app the lookup is created in, and regardless of the user that creates the lookup.

Any ideas on what would cause lookups to fail with these errors? We're on Splunk 6.5.1.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

sjohnson_splunk — Wed, 10 May 2017 20:18:55 GMT

Is there a distsearch.conf in the search app that has a blacklist for lookups?

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

adauria_splunk — Thu, 11 May 2017 13:22:59 GMT

I am the SE assisting here. No, we can't seem to find any distsearch.conf outside of the default directories. I don't see anything in there blacklisting these lookups.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

woodcock — Fri, 12 May 2017 05:19:58 GMT

See the answer by @sjohnson. One way to test is to try this which will work but it will be slower than it should be:

 <search> |lookup local=true dt1 cs_host

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff — Fri, 12 May 2017 12:15:54 GMT

We're not able to find a distsearch.conf file in the search app directory. The query does succeed using local=true.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

adonio — Fri, 12 May 2017 12:43:35 GMT

try and do your search and | lookup dt1.csv cs_host

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

woodcock — Fri, 12 May 2017 16:58:10 GMT

Then you have a permission or (app) scope problem and you must not be running the 2 searches as the same user in the same app. Show me the URL for both |inputlookup dt1, <search> |lookup dt1 cs_host, and <search> |lookup local=true dt1 cs_host and make sure that you are logged in as the same user each time. It is surely that you are in 2 different apps; one which has access to the lookup and the other which does not.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

adauria_splunk — Fri, 12 May 2017 17:00:21 GMT

They were run both as the same regular user, and as the same administrative user, with the same results. Everything was done within the search app only.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff — Fri, 12 May 2017 17:14:08 GMT

Here you are. Hostname and index name removed for privacy.

inputlookup:

<hostname>/en-US/app/search/search?q=%7Cinputlookup%20dt1&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=statistics&sid=1494609014.208099&display.page.search.tab=statistics

index= |lookup dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609057.208159

index= |lookup local=true dt1 cs_host

<hostname>/en-US/app/search/search?q=search%20index%3D<indexname>%20%7Clookup%20local%3Dtrue%20dt1%20cs_host&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now&display.general.type=events&display.page.search.tab=events&sid=1494609075.208172

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

DalJeanis — Fri, 12 May 2017 18:37:22 GMT

Okay, so IIRC, "local" forces the lookup action to be executed on the search head, while the other does not. Doesn't that indicate that the lookup table is not being replicated to the indexers/peers?

https://answers.splunk.com/answers/343835/how-to-distribute-lookup-tables-in-an-indexer-clus.html
https://answers.splunk.com/answers/634/in-a-distributed-search-environment-where-do-my-configurations-go-the-search-head-the-indexers.html

This thread describes a similar issue when the lookup table name was too long to be bundled. (obviously not the case here.)
https://answers.splunk.com/answers/200719/where-does-a-lookup-table-need-to-be-in-a-distribu.html

In any case, on several threads I saw the admonition, "Check your bundle replication error messages." Something in the replication process is screwy.

Lastly, the lesson learned in this one was that the new/altered lookup table had to be on the SH captain, not just any search head.
https://answers.splunk.com/answers/338008/why-do-i-see-old-data-in-my-lookup-table-in-a-sear.html

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

woodcock — Fri, 12 May 2017 22:19:53 GMT

What version of splunk?

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

woodcock — Fri, 12 May 2017 22:20:24 GMT

Are you clustered?

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff — Mon, 15 May 2017 16:52:38 GMT

6.5.1. Search head is standalone. There are multiple indexers.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

dewoodruff — Thu, 22 Jun 2017 15:43:42 GMT

The root cause ended up being a custom app with very long file paths caused the knowledge bundle to fail to replicate. Once the app was removed, the knowledge bundle began replicating again and lookups started working.

The is apparently a known issue with 6.5.1 and has been resolved in some later version.

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

aaraneta_splunk — Thu, 22 Jun 2017 16:03:25 GMT

@dewoodruff - Glad you found the solution to your question. Please don't forget to click "Accept" to close out your question and upvote any answers/comments that were helpful. Thanks!

Re: Why do lookup searches fail with "The lookup table does not exist or is not available" error?

evertonpsp — Tue, 06 Nov 2018 17:47:42 GMT

Helo, Can You help me?

I have a like problem, but i think that the root cause is permission, since the problem happens for the users of a certain Role and the others work.

The file (Lookup table files) is read-only for all users and apps.

The Role Capabilities of the user that works is different from the Role Capabilities of the problem user, but I do not know which one I should add so that both work.

Does anyone have any ideas?
The problem can be Capabilities?

Thanks!