Folder/File ownership by root when Splunk runs as non-root

[dokaas_2]

We have a contractor installing a Splunk instance for us.  For search heads, we have an NVMe volume mounted for the /opt/splunk/var/run folder.  The ./run folder is owned by 'root' and the 'splunk' user cannot write into the folder.

Similar, our indexers have a mounted NVMe volume for the /opt/splunk/var/lib folder and it too is owned by 'root'.  Index folders and files are located one level below that in the ./lib/splunk folder where the 'splunk' user is the owner.

What are the consequences of having 'root' own these folders on the operation of Splunk?  I assumed that when Splunk is running as non-root, that it must be the owner of all folders and files from /opt/splunk on down.  Am I wrong?

[tscroggins]

The primary consequence is invalidation of assumptions made by Splunk during development. Splunk assumes the owner of the Splunk process has read, write, and execute access on $SPLUNK_HOME and all subdirectories. Invalidating this assumption causes unexpected SplunkWeb exceptions at best and splunkd failures at worst.

It may be more appropriate to ask what problem you're trying to solve by making root both the owner of the mount point and the owner of the mounted file system object. If splunk is the owner of /opt/splunk/var, then splunk can delete (unlink) /opt/splunk/var/lib and /opt/splunk/var/run if lib and run are unmounted and empty.

(Edit: As a friendly aside, you're asking your Splunk administrator(s) and the Splunk community, none of whom presumably work for Splunk or have access to Splunk source code, to validate your assumption that everything will be fine if root owns subdirectories of $SPLUNK_HOME. If you don't trust either party to provide a satisfactory answer, I recommend contacting your Splunk technical account manager or Splunk support. Distrust is a healthy aspect of risk mitigation, but find a party you do trust and verify everyone's assumptions.)

[dokaas_2]

This is a new instance being installed by a contractor who posited that it's Linux best practice that root owns all mount points.  Internally, a security architect (not a Splunk admin) supported the 'Linux best practice' argument.  In all of my many years of experience with Splunk, or in training, I have never heard of setting root as the owner of mount points within the /opt/splunk folder tree.  So this is a difference of professional opinion and I'm looking for definitive, authoritative consequences of having root as the owner of the mount points.   Often it takes someone outside the organization to validate an item.

[PickleRick]

That is so wrong and so stupid. If you have a system where each user's home directory is mounted off of a CIFS server on logon (yes, I've seen such systems) should user's home be owned by root? And that's just one (yes, a bit extreme) example.

If that's supposed to be a "best practice", question is "why". Also general "best practice" might not be the best practice for a given situation.

[tscroggins]

@PickleRick keepin' it real.

[tscroggins]

A best practice for my customer isn't a best practice for your customer and vice versa, but in general, if you need to mount a separate file system, I recommend mounting it outside /opt/splunk and either updating Splunk configuration to reference the new location or using symlinks in the /opt/splunk directory tree. The mount point can be owned by root, but once mounted, the file system objects (including the root directory of the mounted file system) should be owned by the splunk user.

[gcusello]

Hi @dokaas_2 ,

as @richgalloway said, Splunk must be the owner of all its objects otherwise it cannor correctly run.

The issue usually is how to permit to a non root user to read root objects in file monitoring or script executing and this is solved in many ways, but all the Splunk folders (executable, run, configurations and data) must be owned by splunk user.

Ciao.

Giuseppe

[dokaas_2]

Thanks.  Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder?  (this is a search head in a cluster)

[richgalloway]

Splunk should be the owner of everything under $SPLUNK_HOME.  Anything else is asking for trouble.  Splunk's inability to access or create a file/directory could manifest itself in a number of unexpected ways.

[dokaas_2]

Thanks. Can you give me any specific issues that might arise if the non-root user 'splunk' can't write into the /opt/splunk/var/run folder? (this is a search head in a cluster)

As well, what about in the /opt/splunk/var/lib folder on an indexer.  Note that buckets are actually stored in /opt/splunk/var/lib/splunk/