Monitoring Splunk

Does splunk capture information for configuration changes

Genti
Splunk Employee
Splunk Employee

is there a way to track configuration changes to splunk - either via splunkweb or command line? The idea is: Lets say i have a saved search. Lets say i login one day and my saved search has been deleted. Is there a way i can find out when it happened, and who (user) deleted it?

This applies to similar situations such as: Who added/deleted a monitor stanza, sourcetypes, etc.. basically is there a way to monitor configuration changes of splunk config files?

tia

.gz

0 Karma
1 Solution

Genti
Splunk Employee
Splunk Employee

This search query should get rid of the issue Lowell mentiones in his comment.

index="_internal" sourcetype="splunkd_access" /servicesNS/*/saved/searches method="DELETE"

View solution in original post

0 Karma

Genti
Splunk Employee
Splunk Employee

This search query should get rid of the issue Lowell mentiones in his comment.

index="_internal" sourcetype="splunkd_access" /servicesNS/*/saved/searches method="DELETE"

0 Karma

Genti
Splunk Employee
Splunk Employee

Actually a better way would be to search for the following:

index="_internal" source="*splunkd_access.log" delete

This, for example will give me a quite nice overview of events:

127.0.0.1 - bubugu [01/Jun/2010:13:45:29.225] "DELETE /servicesNS/bubugu/search/saved/searches/testa HTTP/1.1" 200 2014 - - - 213ms

* host=beefysup01   Options|  
* sourcetype=splunkd_access   Options|  
* source=/splunk/var/log/splunk/splunkd_access.log   Options

Basically, this was a test i tried to see how i could find out deleted saved searches, and i see that the test user "bubugu" was the one who deleted the saved search "testa".

Note, in answers.splunk there is a similar topic, http://answers.splunk.com/questions/2286/search-for-deleted-splunk-users which shows how to find out deleted users.

Both use similar searching syntax. So, to answer my own question the search query needed will be something to the order of:

index="_internal" source="*splunkd_access.log" xxx <- add here more searching terms..

Genti
Splunk Employee
Splunk Employee

Yeap, i found that the hard way.
The first time i ran the search there was only one or two events, soon after the search run there were dozens..

0 Karma

Lowell
Super Champion

Good find. Keep in mind that _index rotates faster than _audit with the default index setup. One thing that gets tricky with either approach is that your own searches start showing up in your query as your refining it. (Because the text of your searches gets logged and then indexed too.)

Lowell
Super Champion

Check the _audit index. I'm not sure what the event will look like exactly, but I would start with a search like this:

index=_audit delete

Just to be clear, splunk audits changes that are made from the UI or CLI, but not changes made directly to the config files themselves. Splunk does create fschange events for $SPLUNK_HOME/etc file changes, but they would no idea what stanza as changed or who is responsible for the change. The content of these files does not get stored, only the fact that the file was changed.

Someone else may be able to fill in some details here, but this should get you started.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...