topic Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file" after upgrade in Splunk Enterprise

Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file" after upgrade

lukasmecir — Wed, 06 Oct 2021 13:04:29 GMT

Hi, I would like to ask for help with following problem:
We have SH cluster (3 nodes) and IDX cluster (3 nodes). We upgraded it from 8.0.9 to 8.1.6 because of EOS of 8.0 version. Everything looks fine, except one thing - sometimes this happens:
I run a search. The search starts, but after a while it stucks (on the line below the place for entering the SPL query, the number of events stops) and after cca 5 minutes the search ends with an error message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file: '/srv/app/int/secmon/splunk/var/run/searchpeers/08270BDA-BE03-4A78-8C6C-95A9CE10BB8D-1633508003/kvstore_s_SA-IdeRjww0FotymhlCIaS1cqkc05a_assetsXy0Y9f6F5lMW4rOy8KLC@P22'"
It happens completely randomly, does not matter what data I search for.
Sometimes this message is generated by only 1 IDX node, sometimes by 2, sometimes by all 3 nodes in IDX cluster.
Error message is always exactly the same (except the part "1633508003", which is time of search).
Sometimes I get partial results (some events returned), sometimes not (0 events returned).
Before upgrade there was no message like this. Could someone help with this? Is it related to the upgrade? And how to fix it? I tried to search through Splunk Community, google around, but did not find anything useful... Thanks in advance.

Lukas Mecir

Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file" after upgrade

jamesmurphy_spl — Thu, 07 Oct 2021 03:43:10 GMT

Hi @lukasmecir, I think you should raise a support case for this issue.

In fixed issues for Splunk 8.1.2, I found this promising note.

2021-01-29 SPL-198149, SPL-199358 KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

See here https://docs.splunk.com/Documentation/Splunk/8.1.2/ReleaseNotes/Fixedissues#Highlighted_issues

but in 8.2.2

2021-05-21 SPL-206067 With large KVstore temporal lookups that are replicated to indexers, turning ON enable_splunkd_kv_lookup_indexing may lead to indexer crash

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_search_head_clustering_issues

So please raise a support case and get the SME's view on how best to address this.

Cheers

Re: Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file"

lukasmecir — Thu, 07 Oct 2021 07:43:49 GMT

Hi @jamesmurphy_spl , thanks for reply. I found the same info you mention in Splunk 8.1.2 fixed issues and it attracted me too. SPL-206067 probably is not the reason, because enable_splunkd_kv_lookup_indexing is set to false in our searchpeers.

Anyway, I raised support case and we'll see...

Cheers

Re: Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file"

jamesmurphy_spl — Thu, 07 Oct 2021 12:37:55 GMT

@lukasmecir

I was referring to setting the value to true in the limits.conf file. See detail

KVStore lookup indexing leads to slow search performance and intermittent errors in searches. In Splunk Enterprise version 8.1.2, if you encounter this problem change the enable_splunkd_kv_lookup_indexing parameter to true in the [lookup] stanza of limits.conf in your $SPLUNK_HOME/etc/system/local directory on your search peers.

but it's perfectly good that you've raised a support case. Fingers crossed you get resolution my friend.

Best

James

Re: Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file"

urbach — Thu, 21 Oct 2021 07:57:20 GMT

Hi @lukasmecir

Have you already got a answer from splunk support? How could you fix this issue?

Thanks and regards

Re: Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file"

amaithani — Thu, 24 Nov 2022 09:29:49 GMT

Please try increasing max_memtable_bytes in limits.conf to higher than default i.e., 25 MB to at least 50MB or more.