We are using Splunk as a security information & event management system. As we review logs or sets of logs, we need to make notes or annotations, to indicate to ourselves and others what we have found, actions we have taken, etc. I am curious how others are doing this, and if there is a good way to do this within Splunk itself? Essentially, this would be using Splunk as a knowledge base.
For example, I could imagine reviewing some traffic on port 8090 on ip 10.1.1.2, and quickly checking to see if we have any notes on this by running a query against a knowledgebase for port=8090 ip=10.1.1.2. And, then, adding notes to it by entering some data in a web form that simply saves the info off into splunk. Another use case I could envision is looking at a log entry, and being able to click on the arrow on the left and have "annotate" as an option, and being able to annotate that log entry. You wouldn't modify the log entry itself (that would be bad), but the knowledgebase would be able to correlate your annotation to the original log entry.