topic Re: Does splunk capture information for configuration changes in Monitoring Splunk

Does splunk capture information for configuration changes

Genti — Wed, 02 Jun 2010 01:59:54 GMT

is there a way to track configuration changes to splunk - either via splunkweb or command line? The idea is: Lets say i have a saved search. Lets say i login one day and my saved search has been deleted. Is there a way i can find out when it happened, and who (user) deleted it?

This applies to similar situations such as: Who added/deleted a monitor stanza, sourcetypes, etc.. basically is there a way to monitor configuration changes of splunk config files?

tia

.gz

Re: Does splunk capture information for configuration changes

Lowell — Wed, 02 Jun 2010 02:07:34 GMT

Check the _audit index. I'm not sure what the event will look like exactly, but I would start with a search like this:

index=_audit delete

Just to be clear, splunk audits changes that are made from the UI or CLI, but not changes made directly to the config files themselves. Splunk does create fschange events for $SPLUNK_HOME/etc file changes, but they would no idea what stanza as changed or who is responsible for the change. The content of these files does not get stored, only the fact that the file was changed.

Someone else may be able to fill in some details here, but this should get you started.

Re: Does splunk capture information for configuration changes

Genti — Wed, 02 Jun 2010 04:32:24 GMT

Actually a better way would be to search for the following:

index="_internal" source="*splunkd_access.log" delete

This, for example will give me a quite nice overview of events:

127.0.0.1 - bubugu [01/Jun/2010:13:45:29.225] "DELETE /servicesNS/bubugu/search/saved/searches/testa HTTP/1.1" 200 2014 - - - 213ms

* host=beefysup01   Options|  
* sourcetype=splunkd_access   Options|  
* source=/splunk/var/log/splunk/splunkd_access.log   Options

Basically, this was a test i tried to see how i could find out deleted saved searches, and i see that the test user "bubugu" was the one who deleted the saved search "testa".

Note, in answers.splunk there is a similar topic, http://answers.splunk.com/questions/2286/search-for-deleted-splunk-users which shows how to find out deleted users.

Both use similar searching syntax. So, to answer my own question the search query needed will be something to the order of:

index="_internal" source="*splunkd_access.log" xxx <- add here more searching terms..

Re: Does splunk capture information for configuration changes

Lowell — Wed, 02 Jun 2010 05:32:35 GMT

Good find. Keep in mind that _index rotates faster than _audit with the default index setup. One thing that gets tricky with either approach is that your own searches start showing up in your query as your refining it. (Because the text of your searches gets logged and then indexed too.)

Re: Does splunk capture information for configuration changes

Genti — Wed, 02 Jun 2010 07:05:13 GMT

Yeap, i found that the hard way.
The first time i ran the search there was only one or two events, soon after the search run there were dozens..

Re: Does splunk capture information for configuration changes

Genti — Tue, 08 Jun 2010 04:03:46 GMT

This search query should get rid of the issue Lowell mentiones in his comment.

index="_internal" sourcetype="splunkd_access" /servicesNS/*/saved/searches method="DELETE"