Hi -
I have a production outage and I am really struggling to fix it - I have had my Unix admin on and Splunk support however they can't pinpoint the issue.
Any help would be amazing, please...I have 1 SH 1 MD and 3 indexers cluster.
On the SH every X minute, i get "waiting for data" and it just hangs there for up to 60 seconds, all the screens 100% unusable.
If i run a search on the MD or Indexers its all fine.We did turn on some verbose logs on in log.cfg
[python]
splunk = DEBUG
And we got this error that i cant really find on google.
2021-10-17 13:07:49,790 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function getServerZoneInfo at 0x7fa655877440>, (), frozenset())
2021-10-17 13:07:49,791 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('j_ycUrLUdujLkZGGP2BX^rH^0aBu4qOEaCTBx89kc4teMYUq_MasVgLLGH9T^lr900ey_UYle7JSPVCMiA1vQiPTChSEQrNiPxobUi1Ut_VKQAwW1UC8H8R8fxXyODmliNS8', <function isModSetup at 0x7fa656048290>, ('Murex',), frozenset())
2021-10-17 13:14:20,027 DEBUG [616c036bfc7fa6567ccfd0] cached:69 - Memoizer expired cachekey=('Qq597goWUdHcWpAp40yLt648IoIOsjngeZUNYmko18k_8LehDC7ZD0Daauwm0vMgbCiNehFe0KYbLHY3m^XFDWqHZPZM1w01V02phdaScMSdDFRrEu8_Q50t1lA5QRBVaiVL6N', <function getEntities at 0x7fa6560a2d40>, ('data/ui/manager',), frozenset({('count', -1), ('namespace', 'Murex')}))
