Following on from an earlier thread where investigations appear in the Analyst Queue for a second and then vanish. That thread was closed with "it's a known issue in 8.4, downgrade to n-1". I'm seeing the identical behaviour on ES 8.3.0, so downgrading is not a way out of it. Sharing what I found in case it helps others, and hoping someone has seen a fix. Environment Splunk Enterprise 10.2.0 Splunk Enterprise Security 8.3.0 3-node Search Head Cluster, multisite indexer cluster Symptom Investigations are created successfully and are fully usable — they open by reference ID, findings attach, notes save. They simply never appear in the Analyst Queue list. Roughly 1 in 4 is affected, consistently, over a 90-day period. What's actually happening The Analyst Queue list is driven by index=mc_investigations. The investigation detail panel reads the mc_incidents KV Store directly. A record only appears in the list if it exists in both places. The affected investigations exist in the KV Store but have no matching event in the index. That is the whole symptom — nothing is deleted, nothing is corrupted. Diagnostic search This gives you the exact records that are invisible:   | inputlookup mc_investigations_lookup | rename _key as id | join type=left id [ search index=mc_investigations earliest=0 | stats count as indexed by id ] | fillnull indexed value=0 | where indexed=0 | table display_id create_time And the overall rate:   | inputlookup mc_investigations_lookup | rename _key as id | join type=left id [ search index=mc_investigations earliest=0 | stats count as indexed by id ] | fillnull indexed value=0 | stats count as created, sum(eval(if(indexed>0,1,0))) as written | eval missing=created-written, miss_pct=round(missing/created*100,1) Note create_time in the lookup is epoch seconds. It is not a failed write — the write is never attempted The index write happens in missioncontrol/bin/blueridge/incidents.py, in a helper that builds a stash file (StashNewWriter) rather than indexing directly. There are three call sites, all gated on the same internal list of "created investigations". That list is only populated when the investigation ID is not already present in the KV Store at the moment update patches are generated. If the record is already persisted at that point, the ID is never added, every write site skips it, and there is no else branch and no log line. No ERROR, no WARN, nothing. Confirmed by log evidence — three investigations created on the same node within 40 minutes: Investigation Creation logged "Saved investigation to mc_investigations index" In index A no entry at all no no B full trace yes yes C full trace yes yes A time-scoped search across all search heads over the failing investigation's creation window returns zero create_incident log events. The failure handler for a genuine write failure logs "Failed to write investigation to mc_investigations index" — that message appears zero times in 30 days. Things I ruled out (so you don't repeat them) Load balancer — reproduced connecting directly to a single search head KV Store replication / recovery — records exist that were written during the claimed failure window get_notable_index macro — at default Missing ES 8.x indexes — all present Multivalue _time in the summary field — visible and invisible records share the same shape Search head serverName misconfiguration and the related "No target_hostname found" redirection error — that error also fires on successful creations, so it's noise here Stash spool backlog — spool directories empty on all nodes, and the per-node distribution of index events matches the per-node distribution of successful creations Workaround for the SOC Until there's a fix, the KV Store lookup returns everything, including the invisible records:   | inputlookup mc_investigations_lookup | sort - create_time I've built a dashboard on that with status/urgency/assignee filters and a column flagging which records are missing from the queue. It's not a fix, but it unblocks analysts completely. Questions Has anyone found a configuration or usage pattern that avoids this? I'm trying to work out whether a particular creation method (e.g. adding findings to a new investigation vs. creating an empty one first) triggers it. Is there a supported way to backfill the missing index records? Writing events into mc_investigations manually seems risky without knowing the exact event shape ES expects. Anyone with a Splunk case on this — has a fix version been named?   ... View more