We are running splunk on SPARC as and account other than root. We have a policy that no applications are allowed to run as root. So we are running it as a user called splunk. We want splunk to read logfiles owned by other users so we have placed the splunk user in the same group that the files have read permissions too. However it appears that splunk only sees it's primary group. Is this really the case? Are there any work arounds to this?
Splunk really doesn't know/care about its group membership. It will call stat() and open() and process the files as long as it doesn't get an EPERM back from the operating system. My first guess is that you need to restart splunkd from a shell that already has these additional groups associated with it. Basically, run the id command from your current shell and make sure that shell has the extra groups, then restart splunkd.
If you su to the 'splunk' user, can you cat the log files in question? You might have permission issues earlier in the path that are preventing access, such as folders that don't have group execute permissions.
If can't figure the permissions out and you are using Solaris 10 (or OpenSolaris), you can add the 'filedacread' privilege to the splunk user account, which would give it access to read every file on the system. Keep in mind the security implications of this, but it is an option.