After Updating to 9.2.2, ESS has stopped creating notables (ESS ver. 7.2.0)

[aluvian]

Hi,

We were using Splunk Enterprise (8.2.5) and ESS (7.2.0) on Debian 12. Everything was working fine until I upgraded Splunk to 9.2.2 (first to 9.1.0 and then to 9.2.2). Next day morning when I checked Security Posture and then Incident Review, I found out that no notable has been created in ESS. Checked the scheduler and correlation searches were run successfully. Also, I tried creating an ad-hoc notable but though Splunk messaged me it was created successfully, I had nothing in my incident review dashboard. Everything else (log ingestion, search, regex...) are working fine. I've been checking the logs for the past few hours but I still have not found anything regarding this issue. I also tried redeploying 7.2.0 but no luck. Any ideas?

[aluvian]

I checked all of them independently and they're all empty.

Running the search in the format you've put throws error : Error in 'inputlookup' command: This command must be the first command of a search.  Which I think is valid as it starts with "inputlookup"

NOTE: Before update, notables were created successfully, so my notables index had data. In order to check if there was any problem with the index itself, I exported notables into CSV files (exporttool) and removed notables index and recreated them.

[aluvian]

I actually did not know about KVStore and the mongodb instance behind it. So I did a little bit of research and try-and-error on that. Disabled KVStore in Splunk and all of the forms and dashboards related to notables and incident review stopped working (threw and error regarding the dashboard not available) So there should be a direct relation between two. Enabled KV and everything went back to normal (except I still have no notables stored)
I've been trying to look for issues in mongo logs but nothing so far. Can you please direct me towards other possible places in KVStore (or similar) to look and investigate? 

[VJD]

Hi,

Check for below, if you get no results, then we can check further.

index=notable

| inputlookup es_notable_events

| inputlookup incident_review_lookup

[VJD]

Hi,

Did you check your kVStore? 

Few lookups are there related to Incident review verify them too.