Thanks to gkapanthy's suggestions I have managed to work through this problem.
The problem again is to tag one or more events in splunk with a change control ticket number, and no singular item tagging currently being available natively in splunk. The basic approach is adding information needed to identify unique events to a lookup, and then using that lookup to enrich regular search results and reverse lookup splunk events based on ticket numbers.
1. Define Lookup
First we define our lookup in transforms.conf and props.conf. I did it in $SPLUNK_HOME/etc/apps/my_app/local but this could also be done in the search app or at the system level.
transforms.conf defines the lookup CSV (in $SPLUNK_HOME/etc/apps/my_app/lookups) and specifies I want to pull more than 1 event out of it (since I want to tag multiple events at times) :
[ticket_correlation]
filename = ticket_correlation.csv
max_matches = 9999999
props.conf applies the lookup to all hosts on my splunk instance. It verifies _cd, index, and splunk_server before spitting out a ticket number:
[host::*]
lookup_tickets = ticket_correlation _cd index splunk_server OUTPUTNEW change_ticket
2. Setup Tagging Workflow Action
Set up a workflow action at the event level. This is easy to do in the Manager UI. Assign a label (I use "Assign Change Ticket: $change_ticket$") and a unique identifier. Show the action in the Event menu and select Action type search. Use the search string below, and save the workflow.
| loadjob $@sid$ events=true
| eval change_ticket="$change_ticket$"
| eval change_ticket = if(isnotnull(change_ticket),change_ticket,"TICKET_MISSING")
| fields - _kv,_meta,_raw,_pre_msg,_s*,_h,_indextime
| fields + _cd, index, splunk_server, change_ticket, host
| inputlookup append=true ticket_correlation
| outputlookup ticket_correlation
Here is what the search does when invoked from an event:
Load the the events from the search the workflow was executed from
Bring over our change_ticket value
Make sure change_ticket is not null. If it is, put a placeholder in the lookup table that we can easily edit later
Remove pesky internal fields we don't want in the lookup table (the want to show up if you include _cd in the | fields +)
Only keep the _cd, index, splunk_server, change_ticket, and host fields
Load any data that currently exists in our ticket_correlation lookup so we can append to it
Output the new lookup table (original lookup table + new events)
3. Tagging Events
Here is a scenario I would use this workflow in:
I receive an alert, a new user account has been created. I run the alert search, and sure enough, a new user account has been created, "Bilbo".
index="foo" sourcetype="WinEventLog:Security" event_id="624" earliest=-1h
I look in our ticketing system, and sure enough there is an approved change request for a new account for new hire "Bilbo". I verify that the only results returned from my current search apply to the change ticket. Hooray, they do: let's tag 'em!
I append the ticket number as a field to the current search:
index="foo" sourcetype="WinEventLog:Security" event_id="624" earliest=-1h | eval change_ticket="12345"
Now I expand the Event menu on an event and select the "Assign Change Ticket: 12345" workflow. A new search is executed that exports the identifying fields for my event(s) to the ticket_correlation lookup table.
4. Enriching Searches and Reporting on Tickets
Since the lookup is applied to every host, events that have been tagged by our workflow action will show up with an additional change_ticket field. We can pull up events based on a specific query:
index=foo change_ticket="12345"
Or we can pull out all tagged events:
index=* change_ticket=*
Now depending on the size of your indexes this will take quite a wile as the _cd field isn't indexed. Chances are you don't have to run searches like this too often (probably only when the auditors are on site).
5. Conclusion
It is definitely possible to utilize
lookups to tag unique events in
splunk. There are some caveats (see
Lowell's comment on gkapanathy's
answer) but it appears to work quite
well. I guess only time will tell if
it'll hold up to production use. If
it doesn't I will revisit this topic
and definitely update it with any
changes I make.
The lookup definition in props.conf
may not be as efficient as possible
(since I am doing it on all hosts).
If this turns out to be a problem I
will make changes and post here.
One pain point is trying to remember
to add the eval change_ticket=xxx to
events I want to save but it is a lot
easier to do than to type up the
inputlookup/outputlookup commands
every time. I am also mostly
unfamiliar with Python and didn't
want to learn it to solve this
problem (plus this way this whole
setup should technically be
supportable by splunk!)
Don't forget to make the workflow action public, else only you will be able to use it! (which may be desirable)
If anybody comes up with a more
elegant or user friendly solution
please do post it here as I am
definitely interested in it.
... View more