topic Re: Monitoring Splunk logs for alert delete in Alerting

Monitoring Splunk logs for alert delete

snehal8 — Wed, 11 Feb 2015 14:15:24 GMT

Hello All,

We have two search head, three indexers clustered, a cluster master, and a deployment server. All running Windows 2k8 R2.

We are finding some alerts are deleted form search head, and need to investigate log files to monitor which user deleted, when and so on.

Cloud you please guide me how to figure out this?

Thanks.

Re: Monitoring Splunk logs for alert delete

aakwah — Mon, 28 Sep 2020 18:56:34 GMT

Hello,

All audit information are saved on audit.log /opt/splunk/var/log/splunk/audit.log audit logs are indexed in _audit index, you can search them index=_audit and create alerts on search queries you want.

Regards

Re: Monitoring Splunk logs for alert delete

snehal8 — Wed, 11 Feb 2015 16:06:59 GMT

Thanks for reply @aakwah, but when i searched for index=_audit "mysearch name" "*delete*" its displaying log for my this search string, please guide me if am going wrong ?

Re: Monitoring Splunk logs for alert delete

aakwah — Mon, 28 Sep 2020 18:56:43 GMT

Welcome, you can make use of action field to specify a certain action you want to look for,

index=_audit action=delete

Some of avialble actions:

index=_audit | stats count by action | table action

CREATE_PASSWORD
EDIT_PASSWORD
GET_PASSWORD
REMOVE_PASSWORD
accelerate_datamodel
accelerate_search
add
delete
edit_roles
edit_server
edit_user
embed_report
license_edit
list_inputs
login attempt
quota
read_session_token
rest_properties_get
rest_properties_set
restart_splunkd

Re: Monitoring Splunk logs for alert delete

lmyrefelt — Wed, 11 Feb 2015 19:56:27 GMT

Remember to check so you are not over-writing any files under local or similar ... if you are seemingly "loosing" data , searches, alerts, views, whatever ... this can happen thru the forwarder-managment / deployment server / cluster-deploy tools that are in use.

Re: Monitoring Splunk logs for alert delete

MuS — Thu, 12 Feb 2015 09:43:37 GMT

Hi snehal8,

run this search:

index=_audit host=YourHostName action=alert_deleted

the result will look like this:

Audit:[timestamp=02-12-2015 10:39:21.783, user=TheBadGuyHowDeletedTheAlert, action=alert_deleted, sid="scheduler__AnyUserName_REFfUkNQX0xEQVA__RMD5d4292166408c9a03_at_1423733700_18910", trigger_time=1423733705, deleted=1][n/a]

Hope this helps ...

cheers, MuS

Re: Monitoring Splunk logs for alert delete

karlbosanquet — Wed, 17 Feb 2016 21:53:59 GMT

https://answers.splunk.com/answers/43339/alert-on-deleted-data.html

Specifically;

Put a crazy string in your search, like so:

index=_audit "action=search" search=*delete* NOT dfsdkljwkwtw
This will prevent your search from showing up in the results.

You might want to refine it a big using a regex to look for | delete, |delete, | delete , etc.