Splunk generates alerts, for example if a server fails to ping or isn't running required services. Is it possible to manually add information to incidents tripped in Splunk's alert manager, in order to correlate Splunk alerts with incident or outage information?
If we have an alert, I would like for one of our admins to be required to document exactly what happened. Could tags be used for that?
not completely sure what your situation is, but you could tag the events involved in the alert via Splunk Web. then you can search on the tags for future analysis. you could define standard tags for different incidents or outages, or even for certain types of incidents and outages for use in future situations.
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Abouttagsandaliases