Splunk Enterprise

Message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file" after upgrade

lukasmecir
Path Finder

Hi, I would like to ask for help with following problem:
We have SH cluster (3 nodes) and IDX cluster (3 nodes). We upgraded it from 8.0.9 to 8.1.6 because of EOS of 8.0 version. Everything looks fine, except one thing - sometimes this happens:
I run a search. The search starts, but after a while it stucks (on the line below the place for entering the SPL query, the number of events stops) and after cca 5 minutes the search ends with an error message "Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file: '/srv/app/int/secmon/splunk/var/run/searchpeers/08270BDA-BE03-4A78-8C6C-95A9CE10BB8D-1633508003/kvstore_s_SA-IdeRjww0FotymhlCIaS1cqkc05a_assetsXy0Y9f6F5lMW4rOy8KLC@P22'"
It happens completely randomly, does not matter what data I search for.
Sometimes this message is generated by only 1 IDX node, sometimes by 2, sometimes by all 3 nodes in IDX cluster.
Error message is always exactly the same (except the part "1633508003", which is time of search).
Sometimes I get partial results (some events returned), sometimes not (0 events returned).
Before upgrade there was no message like this. Could someone help with this? Is it related to the upgrade? And how to fix it? I tried to search through Splunk Community, google around, but did not find anything useful... Thanks in advance.

Lukas Mecir

Labels (2)
0 Karma
1 Solution

jamesmurphy_spl
Splunk Employee
Splunk Employee

Hi @lukasmecir, I think you should raise a support case for this issue. 

In fixed issues for Splunk 8.1.2, I found this promising note.

2021-01-29 SPL-198149, SPL-199358 KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

See here https://docs.splunk.com/Documentation/Splunk/8.1.2/ReleaseNotes/Fixedissues#Highlighted_issues

but in 8.2.2 

2021-05-21 SPL-206067 With large KVstore temporal lookups that are replicated to indexers, turning ON enable_splunkd_kv_lookup_indexing may lead to indexer crash

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_s...

So please raise a support case and get the SME's view on how best to address this.

Cheers

View solution in original post

0 Karma

amaithani
Splunk Employee
Splunk Employee

Please try increasing max_memtable_bytes in limits.conf to higher than default i.e., 25 MB to at least 50MB or more.

0 Karma

lukasmecir
Path Finder

Hi @jamesmurphy_spl , thanks for reply. I found the same info you mention in Splunk 8.1.2 fixed issues and it attracted me too. SPL-206067 probably is not the reason, because enable_splunkd_kv_lookup_indexing is set to false in our  searchpeers.

Anyway, I raised support case and we'll see...

Cheers

0 Karma

urbach
Explorer

Hi @lukasmecir 

Have you already got a answer from splunk support? How could you fix this issue?

Thanks and regards

0 Karma

jamesmurphy_spl
Splunk Employee
Splunk Employee

@lukasmecir 

I was referring to setting the value to true in the limits.conf file. See detail

KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

In Splunk Enterprise version 8.1.2, if you encounter this problem change the enable_splunkd_kv_lookup_indexing parameter to true in the [lookup] stanza of limits.conf in your $SPLUNK_HOME/etc/system/local directory on your search peers.

but it's perfectly good that you've raised a support case. Fingers crossed you get resolution my friend.

 

Best 

James

jamesmurphy_spl
Splunk Employee
Splunk Employee

Hi @lukasmecir, I think you should raise a support case for this issue. 

In fixed issues for Splunk 8.1.2, I found this promising note.

2021-01-29 SPL-198149, SPL-199358 KVStore lookup indexing leads to slow search performance and intermittent errors in searches.

See here https://docs.splunk.com/Documentation/Splunk/8.1.2/ReleaseNotes/Fixedissues#Highlighted_issues

but in 8.2.2 

2021-05-21 SPL-206067 With large KVstore temporal lookups that are replicated to indexers, turning ON enable_splunkd_kv_lookup_indexing may lead to indexer crash

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_s...

So please raise a support case and get the SME's view on how best to address this.

Cheers

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...