Splunk Search

Error in 'lookup' command: Why is it failing to re-open lookup file?

halu
Loves-to-Learn Lots

Hello Splunker!

 

Sometimes my searches on Splunk Enterprise Security Search Head ran into following error (mostly) without any results, sometimes there are only a view results :

 

[idx1, idx4 ...] Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file: '/opt/splunk/var/run/searchpeers/splunksearchhead-1631016538/kvstore_s_SA-IdeRjww0FotymhlCIaS1cqkc05a_identix3UXVbINERGdyPwDBuI5US7E'.

 

Sometimes the searches work, somtimes they not. There is also a "normal" Splunk Search Head, the same search works all the time. If the error appears also the Incident Review needs about 

I already checked the bundle size of both Search Heads and the ES bundle is about 800mb. The "normal" Search Head bundle is  about 1,1gb.

 

Splunk Enterprise 8.2.1

Splunk Enterprise Security 6.6.0

Splunk Cluster with 2 sites, each site 8 idx.

 

I would greatly appreciate any help 🙂

 

 

 

Labels (1)
0 Karma

ljramv
Explorer

Hi, anybody managed to resolve this? Got a ticket open with Splunk for a month now...

0 Karma

satishkompelly
New Member

@halu I'm getting the error and the searches are running and getting paused and resume after sometime we also upgraded to 8.2.3 recently, did you find any troubleshooting's to prevent this error.

Satish

0 Karma

sikder
Observer

Hello,

I have the same issue after upgrading to Splunk 8.2.3. Did you solve the issue?

 

0 Karma

halu
Loves-to-Learn Lots

Sorry no I didn't resolv it yet. The error didn't appear 1-2 weeks, but its still happen user ran into this error

0 Karma

amaithani
Splunk Employee
Splunk Employee

Please try increasing the max_memtable_bytes to higher than default i.e., 25 MB to 50MB.

 

0 Karma

ljramv
Explorer

In the ES correlation set up page for Asset and Identity management ensure you have "enable selectively for source types" if used or disable if you don't.

0 Karma

wmuselle
Path Finder

could you elaborate on your answer.

The message seems replication bundle related (replication=true for kv store collection).

What is the link with your answer ?

 

thanks in advance

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...