forwarder fails to start with permission errors

[swissarmychains]

Brand new system, new colo.

cat /etc/redhat-release 

CentOS release 6.5 (Final)

arch

x86_64

Installing this:

splunkforwarder-6.1.1-207789-Linux-x86_64.tgz

After the unzip, attempting to start the forwarder:

root> /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt

Gives all these "Permission Denied" errors.
Even when changing the file:
/opt/splunkforwarder/etc/users/users.ini
to 777 it still gives this error.

Help!

./splunk start

Splunk> Winning the War on Error

Checking prerequisites...
    Checking mgmt port [8089]: open
Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
    Checking conf files for problems...
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
Cannot open file to check: /opt/splunkforwarder/etc/system/local/inputs.conf
        Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini

FYI:

./bin/splunk btool check --debug
Checking: /opt/splunkforwarder/etc/apps/search/local/inputs.conf
No spec file for: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/server.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/web.conf
No spec file for: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf
No spec file for: /opt/splunkforwarder/etc/apps/search/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/restmap.conf
Checking: /opt/splunkforwarder/etc/system/default/alert_actions.conf
No spec file for: /opt/splunkforwarder/etc/system/default/app.conf
Checking: /opt/splunkforwarder/etc/system/default/audit.conf
Checking: /opt/splunkforwarder/etc/system/default/authentication.conf
Checking: /opt/splunkforwarder/etc/system/default/authorize.conf
No spec file for: /opt/splunkforwarder/etc/system/default/conf.conf
Checking: /opt/splunkforwarder/etc/system/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/system/default/inputs.conf
Checking: /opt/splunkforwarder/etc/system/default/limits.conf
Checking: /opt/splunkforwarder/etc/system/default/outputs.conf
Checking: /opt/splunkforwarder/etc/system/default/procmon-filters.conf
Checking: /opt/splunkforwarder/etc/system/default/props.conf
Checking: /opt/splunkforwarder/etc/system/default/restmap.conf
Checking: /opt/splunkforwarder/etc/system/default/server.conf
Checking: /opt/splunkforwarder/etc/system/default/source-classifier.conf
Checking: /opt/splunkforwarder/etc/system/default/web.conf
Checking: /opt/splunkforwarder/etc/system/local/inputs.conf
Checking: /opt/splunkforwarder/etc/system/local/server.conf

[grijhwani]

For starters, being on CentOS why you chose not to install the x86_64 RPM version. The install script will create the right user/and group for you. Installing with the tarball needs manual intervention to get things right, I fail to see any benefit from side-stepping, and you may your problem rooted in this very fact. As a permissions issue this looks more like an underlying system/file-system-level problem.

First of all, your one example prompt says "root>", but does than mean you genuinely are running su? Or is it just a static prompt? Second, what is the state of the files? You say you changed permissions to 777, but that's a really awful thing to do. Show us the result of

id
ls -ld /opt/splunkforwarder/etc/users/users.ini

Can you

cat /opt/splunkforwarder/etc/users/users.ini

Are you running SELinux enabled? Is it enforcing? If so (and I suspect this is probably your problem) your security context is going to be out of whack. It may be that that is disallowing the file open.