Access Finding/Investigation notes in ES 8

[ljvc]

Hi there,

we're currently migrating to ES 8 and need to see Work Notes (comments) provided by analysts in some dashboards/reports. Previously, the incident_updates_lookup contained the "comment" field, which held this information, and was easy to access in a search.

With ES 8, this was obviously mentioned as a limitation - "The Comments feature available in prior versions of Splunk Enterprise Security is now replaced by an enhanced capability to add notes."

How can we access those notes (KV Store/Lookup/...) outside of having to click through the Mission Control/Analyst Queue manually? Where are they stored?

[ljvc]

As usual, I figured it out shortly after finally asking.
Notes are kept in the mc_notes collection in the missioncontrol app, if anyone else was wondering...

[randoj]

Good afternoon @ljvc. Could you provide some direction on how you're accessing the mc_notes collection from within the Mission Control app? Struggling to find this.

[ljvc]

Hi @randoj ! 

We just created a lookup definition manually in a local/transforms.conf, as you would with any other KV Store lookup.
Additionally, we needed to do the same for the mc_incidents collection, as it is needed to correlate notable_ids and incident_ids, the latter of which are used in mc_notes.

It probably is easier to access the collections using the Python SDK and scripts, but this solution worked for us and required less setup.

Hope this helps!

[randoj]

@ljvc I appreciate the information, this is helpful. Would you be able to share your transforms.conf files against the mc_incidents and mc_notes collections? I'd like to better understand how that correlation with incident_id is happening between the collections and dumping it into something readable in a lookup.

[ljvc]

@randoj unfortunately, I cannot share the exact files. However, you should be able to get the incident id for each finding using its calculated rule_id (compare the eval statement for rule_id/event_id in [Incident Review - Main] in SA-ThreatIntelligence/default/savedsearches.conf) via the mc_incidents collection, which has a field notable_id iirc. Then, use that id as a key against the mc_notes collection, and you can get notes for findings. Hope this clears things up a bit!

[randoj]

@ljvc I appreciate the information you were able to provide, this is helpful. On a side note I do have an active case open with Splunk support on this topic. Their latest update was that this has been a reported issue, and that they expect it to be addressed in ES 8.2 per an internal JIRA ticket.

[adrezende_splun]

Download Splunk App for Lookup File Editing app. Nn Lookups menu, select All and search for mc_notes. On Actions menu, click the magnifier button to search the mc_notes lookup. A prompt will show up asking you to create a lookup transform. Add the name that you want and click Create transform.

Open a new search and search | inputlookup mc_notes to show mv_notes content.

[hcpr]

Hi, 

Following up on the above discussion, has anyone else discovered that there are quite a few instances where the "incident_id" field is blank in the mc_notes lookup?
The other fields (autor.username, create_time and content) contain the correct information but there is nothing in incident_id.

Makes it a bit difficult to match the note to the corresponding incident 🙂

[ljvc · Accepted solution]

Hello hcpr,

we did run into the same issue shortly after my previous post, and I forgot to give an update in here.

The app "missioncontrol" exposes a few endpoints to do with incidents and investigations, and tracing the behavior of Enterprise Security when fetching comments led us to its OpenAPI spec which you can find at missioncontrol/mcopenapi.yaml.

The incidents endpoint, when fed a finding/notable ID, will return a list of comments. I recommend everyone to take a look at the requests in your browser's developer tools when interacting with finding notes on the Analyst Queue to see how the endpoint works. Ultimately, we went that way and implemented a custom command to perform the same requests at search time. This is now working flawlessly for us so far, getting even those notes which have no incident_id or source in mc_notes.

Hope this helps!