I need to perform forensic analysis on compromised computers, but they are sometimes not online anymore by the time I've seen the AV alert and have Encase up and running. It would be extremely helpful if Splunk could search for that computer (ping, Windows Security events, whatever) email me when the appropriate traffic is observed, and then stop the search.
Detecting traffic from the computer and sending the email is no problem - it's how to get the search to stop once the target computer is online that has me stumped. Ideally, I would type something like 'sourcetype="IsOnline" hostname="HackedPC" notify="MyEmail@company.com"', and the search would run until I get the email, without my having to stop it manually.
Thanks!
One approach would be to set the alert action for your search to a script.
The script can then send you an email and also interact with Splunk via the REST API.
http://dev.splunk.com/view/SP-CAAADQ5 provides an example for disabling a search via the API:
curl -k -u admin:changeme -d "disabled=1" -d 'search="fail+OR+error"' https://localhost:8089/servicesNS/admin/search/saved/searches/web_errors
You also create, delete and otherwise manipulate saved searches without having to interact with the UI.
It's also possible just disable the alert action.