Is there any way to securely disable the delete command in Splunk?

[hettervik]

In Splunk there exist a delete command. Any admin in Splunk can give themself the capability to use this command. In theory, if a single admin user in our Splunk environment is compromised, the attacker can delete all data from the Splunk indexers. I know that the data is not actually deleted from disk when using the delete command, but still it is for all practical purposes deleted.

Is there any way to securely disable the delete command/capability in Splunk, so that not even administrators can get access to it?

Preferably we want to disable the command on the indexer layer, so that even if the OS on the server hosting the search head is compromised the command cannot be used. Alternatively, if the command can be disabled on the search head in a way that it cannot be re-enabled through the web interface, that is better than nothing.

[hettervik_new]

Okay, update on this. I've been looking into federated search, and realize that one can use this to guarantee that the users cannot delete data on the Splunk indexers, and more.

Let me explain. If you have a Splunk deployment with a search head, let's call it "main SH", then you can set up an extra search head, let's call it "external SH". You could then give access to users, all or only the "risky" ones, on the external SH, and set up a federated search option from the external SH to the main SH. Then, even if a user on the external SH gets full admin rights, and even if they are able to get root on the external SH server, they still would not be able to elevate their privileges on the main SH.

This is kind of an extreme solution, that adds a lot of complexity to your Splunk deployment. Preferably, if there was a way to disable the "can_delete" capability on the SH without being able to re-add it through the Splunk web, that would have been easier, but as of now this seems to be the only way.

If someone agrees that there should be an option to safely remove the "can_delete" capability, feel free to give my post in Splunk Ideas a thumb up: https://ideas.splunk.com/ideas/EID-I-1687

[gcusello]

Hi @hettervik,

for my knowledge it isn't possible to disble the delete command, but even if you can disable it, having an admin grant, you can clean an entire index by CLI, and it's possible to rubber all data.

For this reason, i think that it's useless.

I hint to protect the access to your system in a different way: MFA,  PAM, etc...

Ciao.

Giuseppe

[hettervik]

Thanks. Though we have a distributed environment, so if the attacker gets access to CLI on the search head, they still cannot delete the data on the indexers. We have extra layers of protection for the indexers. Therefore it would still help to be able to disable the delete command, if I'm not missing something.

[starcher]

Just an FYI delete command does not delete. It only hides events from search. If you are knowledgable enough you can edit files on disk at the indexer to bring it back.

Removing rights to run the command and monitoring for it's use is generally enough.

[hettervik_new]

Yes, I know, but say someone indeed does run the delete command on all data, it could still create some a fair amount of downtime before the Splunk admins are able to figure out what's wrong and restore all the data. If somebody deletes data say before a weekend or a holiday, the downtime would be even greater.

Also, I'm aware that normally the admin rights are needed to access the delete command, but in my Splunk environment the delete command is basically never needed, so it adds no benefit, but adds a risk. I'm guessing this is the case for a lot of other customers as well. Thus, removing the option completely from the search head would be the best and most secure solution.

[gcusello]

Hi @hettervik,

as I said there isn't any way to disable the delete command and my hint is to find a different way to protect the access to your systems (e.g. using PAM).

Maybe you could ask to Splunk Professional Services to make an intervene to disable the delete command using methods not accessible to us common mortals!

Ciao.

Giuseppe