Production issue - Memoizer expired - Any ideas?

[robertlynch2020]

Hi -

I have a production outage and I am really struggling to fix it - I have had my Unix admin on and Splunk support however they can't pinpoint the issue.

Any help would be amazing, please...I have 1 SH 1 MD and 3 indexers cluster.

On the SH every X minute, i get "waiting for data" and it just hangs there for up to 60 seconds, all the screens 100% unusable.
If i run a search on the MD or Indexers its all fine.We did turn on some verbose logs on in log.cfg
[python]
splunk = DEBUG

And we got this error that i cant really find on google.

2021-10-17 13:07:49,790 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function getServerZoneInfo at 0x7fa655877440>, (), frozenset())
 2021-10-17 13:07:49,791 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function isModSetup at 0x7fa656048290>, ('Murex',), frozenset())
2021-10-17 13:14:20,027 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('Qq597goWUdHcWpAp40yLt648IoIOsjngeZUNYmko18k_8LehDC7ZD0Daauwm0vMgbCiNehFe0KYbLHY3m^XFDWqHZPZM1w01V02phdaScMSdDFRrEu8_Q50t1lA5QRBVaiVL6N', <function getEntities at 0x7fa6560a2d40>, ('data/ui/manager',), frozenset({('count', -1), ('namespace', 'Murex')}))

6:50

Any help would be so good, as i am in the process of thinking of installing the apps onto the search head for tomorrow Production, i think that will work?

[dwickram]

@robertlynch2020  Hi there, I found couple of resources for the error message but not sure if they would help, but may lead to isolate the issue. 

https://www.tek271.com/software/java/memoizer

https://www.baeldung.com/guava-memoizer

Since it's a single SH,  are you doing any API pulls / DB Connect sort of mechanism to pull the data in? Was there any recent app installation / upgrade happened?  I would disable above API/ DB Connect inputs one by one and isolate the issue.  Are these error messages from splunkd.log?