Splunk Search

Production issue - Memoizer expired - Any ideas?

robertlynch2020
Motivator
Hi -
 
I have a production outage and I am really struggling to fix it - I have had my Unix admin on and Splunk support however they can't pinpoint the issue.
 
Any help would be amazing, please...I have 1 SH 1 MD and 3 indexers cluster.
 
On the SH every X minute, i get "waiting for data" and it just hangs there for up to 60 seconds, all the screens 100% unusable.
If i run a search on the MD or Indexers its all fine.We did turn on some verbose logs on in log.cfg
[python]
splunk = DEBUG
 
And we got this error that i cant really find on google.
 
2021-10-17 13:07:49,790 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function getServerZoneInfo at 0x7fa655877440>, (), frozenset())
 2021-10-17 13:07:49,791 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function isModSetup at 0x7fa656048290>, ('Murex',), frozenset())
2021-10-17 13:14:20,027 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('Qq597goWUdHcWpAp40yLt648IoIOsjngeZUNYmko18k_8LehDC7ZD0Daauwm0vMgbCiNehFe0KYbLHY3m^XFDWqHZPZM1w01V02phdaScMSdDFRrEu8_Q50t1lA5QRBVaiVL6N', <function getEntities at 0x7fa6560a2d40>, ('data/ui/manager',), frozenset({('count', -1), ('namespace', 'Murex')}))
Any help would be so good, as i am in the process of thinking of installing the apps onto the search head for tomorrow Production, i think that will work?
Labels (1)
Tags (1)
0 Karma

dwickram
Explorer

@robertlynch2020  Hi there, I found couple of resources for the error message but not sure if they would help, but may lead to isolate the issue. 

https://www.tek271.com/software/java/memoizer

https://www.baeldung.com/guava-memoizer

Since it's a single SH,  are you doing any API pulls / DB Connect sort of mechanism to pull the data in? Was there any recent app installation / upgrade happened?  I would disable above API/ DB Connect inputs one by one and isolate the issue.  Are these error messages from splunkd.log? 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...