We have two search head, three indexers clustered, a cluster master, and a deployment server. All running Windows 2k8 R2.
We are finding some alerts are deleted form search head, and need to investigate log files to monitor which user deleted, when and so on.
Cloud you please guide me how to figure out this?
run this search:
index=_audit host=YourHostName action=alert_deleted
the result will look like this:
Audit:[timestamp=02-12-2015 10:39:21.783, user=TheBadGuyHowDeletedTheAlert, action=alert_deleted, sid="scheduler__AnyUserName_REFfUkNQX0xEQVA__RMD5d4292166408c9a03_at_1423733700_18910", trigger_time=1423733705, deleted=1][n/a]
Hope this helps ...
Remember to check so you are not over-writing any files under local or similar ... if you are seemingly "loosing" data , searches, alerts, views, whatever ... this can happen thru the forwarder-managment / deployment server / cluster-deploy tools that are in use.
All audit information are saved on audit.log
/opt/splunk/var/log/splunk/audit.log audit logs are indexed in _audit index, you can search them
index=_audit and create alerts on search queries you want.
Put a crazy string in your search, like so:
index=_audit "action=search" search=*delete* NOT dfsdkljwkwtw
This will prevent your search from showing up in the results.
You might want to refine it a big using a regex to look for | delete, |delete, | delete , etc.
Welcome, you can make use of action field to specify a certain action you want to look for,
Some of avialble actions:
index=_audit | stats count by action | table action