Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file" after upgrade

[lukasmecir]

Hi, I would like to ask for help with following problem:
We have SH cluster (3 nodes) and IDX cluster (3 nodes). We upgraded it from 8.0.9 to 8.1.6 because of EOS of 8.0 version. Everything looks fine, except one thing - sometimes this happens:
I run a search. The search starts, but after a while it stucks (on the line below the place for entering the SPL query, the number of events stops) and after cca 5 minutes the search ends with an error message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file: '/srv/app/int/secmon/splunk/var/run/searchpeers/08270BDA-BE03-4A78-8C6C-95A9CE10BB8D-1633508003/kvstore_s_SA-IdeRjww0FotymhlCIaS1cqkc05a_assetsXy0Y9f6F5lMW4rOy8KLC@P22'"
It happens completely randomly, does not matter what data I search for.
Sometimes this message is generated by only 1 IDX node, sometimes by 2, sometimes by all 3 nodes in IDX cluster.
Error message is always exactly the same (except the part "1633508003", which is time of search).
Sometimes I get partial results (some events returned), sometimes not (0 events returned).
Before upgrade there was no message like this. Could someone help with this? Is it related to the upgrade? And how to fix it? I tried to search through Splunk Community, google around, but did not find anything useful... Thanks in advance.

Lukas Mecir

[amaithani]

Please try increasing max_memtable_bytes in limits.conf to higher than default i.e., 25 MB to at least 50MB or more.

[lukasmecir]

Hi @jamesmurphy_spl , thanks for reply. I found the same info you mention in Splunk 8.1.2 fixed issues and it attracted me too. SPL-206067 probably is not the reason, because enable_splunkd_kv_lookup_indexing is set to false in our  searchpeers.

Anyway, I raised support case and we'll see...

Cheers

[urbach]

Hi @lukasmecir 

Have you already got a answer from splunk support? How could you fix this issue?

Thanks and regards

[jamesmurphy_spl]

@lukasmecir 

I was referring to setting the value to true in the limits.conf file. See detail

KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

In Splunk Enterprise version 8.1.2, if you encounter this problem change the enable_splunkd_kv_lookup_indexing parameter to true in the [lookup] stanza of limits.conf in your $SPLUNK_HOME/etc/system/local directory on your search peers.

but it's perfectly good that you've raised a support case. Fingers crossed you get resolution my friend.

Best 

James

[jamesmurphy_spl · Accepted solution]

Hi @lukasmecir, I think you should raise a support case for this issue. 

In fixed issues for Splunk 8.1.2, I found this promising note.

2021-01-29 SPL-198149, SPL-199358 KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

See here https://docs.splunk.com/Documentation/Splunk/8.1.2/ReleaseNotes/Fixedissues#Highlighted_issues

but in 8.2.2 

2021-05-21 SPL-206067 With large KVstore temporal lookups that are replicated to indexers, turning ON enable_splunkd_kv_lookup_indexing may lead to indexer crash

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_s...

So please raise a support case and get the SME's view on how best to address this.

Cheers