A way to audit changes to savedsearches.conf

[stillfree]

Hi All,

It's already been determined that alarms/reports modifications are not being audited in _audit and _internal indexes. Reference to the links below.  @richgalloway can you recall the topic you gave me an answer to last week (second link below)? 

https://ideas.splunk.com/ideas/E-I-49 

https://community.splunk.com/t5/Alerting/Audit-changes-modifications-to-Alerts-Reports-using-REST-AP... 

Apart from keeping alerts/reports confs in Change Management System, the only option is (please correct me if i'm wrong) to use the bellow search that utilizes REST API and this is going to give us 1) time of modification 2) name of the alert/report in the form of uri and we can play further to extract this as a separate field and 3) who done it. 

index="_internal" sourcetype="splunkd_ui_access" servicesNS file!="notify" method=POST

As you can see from above this would pick up changes made ONLY through the GUI but how about the CLI. If someone tampered with the savedsearches.conf file I'd like to audit those changes somewhere.

Is there a straightforward way? I was thinking of file monitor of the file in Splunk and raise an alarm when something is changed but can't think of how I'll write the search query for the alarm. As am I novice to Splunk I read an article about diff command but found the documentation not clear and couldn't fully grasp how should I use it and if it's appropriate for my goal here?

Also I read somewhere about putting the file in version control and monitor the changes but I hihgly doupt it that our customer would agree to that approach (don't ask).

Any suggestions would be much appreciated.

Thank you in advance. 

[richgalloway · Accepted solution]

That query is interesting.  I ran it in my sandbox and it returned 75 results over 15 minutes even though I made no changes to anything (I had only run searches during that time).  I expect you will get a lot of false positives.

The same can be said for monitoring the savedsearches.conf file, although you should see a lot fewer of them.  A new saved search will trigger a change alert and someone will have to investigate if the change is a new search or a modification of an existing one.

I have not used the SPL diff command, but it's seem a lot like the Linux diff command.  That command does not work well with Splunk .conf files because of the way Splunk updates them.  When a stanza changes, even if it's a single character, the entire stanza is removed and the updated version is written to the end of the file.  A before-and-after comparison usually shows several lines deleted and several added.  It then falls on a human to determine the nature of that change.

[stillfree]

I agree about the false positive and you are correct. I can confirm that displays changes made to KO's as I also did a test on my own instance but as you said it gives back way too much FPs. I guess some further filtering would narrow down the noise. Thank you for your reply.