Getting Data In

forwarder fails to start with permission errors

swissarmychains
New Member

Brand new system, new colo.


cat /etc/redhat-release
CentOS release 6.5 (Final)
arch
x86_64

Installing this:


splunkforwarder-6.1.1-207789-Linux-x86_64.tgz

After the unzip, attempting to start the forwarder:


root> /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt

Gives all these "Permission Denied" errors.
Even when changing the file:
/opt/splunkforwarder/etc/users/users.ini
to 777 it still gives this error.

Help!

./splunk start

Splunk> Winning the War on Error

Checking prerequisites...
    Checking mgmt port [8089]: open
Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
ERROR - Error opening "/opt/splunkforwarder/var/log/splunk/splunkd-utility.log": Permission denied
    Checking conf files for problems...
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
Cannot open file to check: /opt/splunkforwarder/etc/system/local/inputs.conf
        Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini
ERROR UsernameMapper - Cannot create username mapping file: /opt/splunkforwarder/etc/users/users.ini: Permission denied
ERROR IniFile - Cannot open file=/opt/splunkforwarder/etc/users/users.ini for parsing: Permission denied
ERROR UsernameMapper - Error opening username mapping file: /opt/splunkforwarder/etc/users/users.ini

FYI:

./bin/splunk btool check --debug
Checking: /opt/splunkforwarder/etc/apps/search/local/inputs.conf
No spec file for: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/server.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/web.conf
No spec file for: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf
No spec file for: /opt/splunkforwarder/etc/apps/search/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/restmap.conf
Checking: /opt/splunkforwarder/etc/system/default/alert_actions.conf
No spec file for: /opt/splunkforwarder/etc/system/default/app.conf
Checking: /opt/splunkforwarder/etc/system/default/audit.conf
Checking: /opt/splunkforwarder/etc/system/default/authentication.conf
Checking: /opt/splunkforwarder/etc/system/default/authorize.conf
No spec file for: /opt/splunkforwarder/etc/system/default/conf.conf
Checking: /opt/splunkforwarder/etc/system/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/system/default/inputs.conf
Checking: /opt/splunkforwarder/etc/system/default/limits.conf
Checking: /opt/splunkforwarder/etc/system/default/outputs.conf
Checking: /opt/splunkforwarder/etc/system/default/procmon-filters.conf
Checking: /opt/splunkforwarder/etc/system/default/props.conf
Checking: /opt/splunkforwarder/etc/system/default/restmap.conf
Checking: /opt/splunkforwarder/etc/system/default/server.conf
Checking: /opt/splunkforwarder/etc/system/default/source-classifier.conf
Checking: /opt/splunkforwarder/etc/system/default/web.conf
Checking: /opt/splunkforwarder/etc/system/local/inputs.conf
Checking: /opt/splunkforwarder/etc/system/local/server.conf
0 Karma

grijhwani
Motivator

For starters, being on CentOS why you chose not to install the x86_64 RPM version. The install script will create the right user/and group for you. Installing with the tarball needs manual intervention to get things right, I fail to see any benefit from side-stepping, and you may your problem rooted in this very fact. As a permissions issue this looks more like an underlying system/file-system-level problem.

First of all, your one example prompt says "root>", but does than mean you genuinely are running su? Or is it just a static prompt? Second, what is the state of the files? You say you changed permissions to 777, but that's a really awful thing to do. Show us the result of

id
ls -ld /opt/splunkforwarder/etc/users/users.ini

Can you

cat /opt/splunkforwarder/etc/users/users.ini

Are you running SELinux enabled? Is it enforcing? If so (and I suspect this is probably your problem) your security context is going to be out of whack. It may be that that is disallowing the file open.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...