On installing the universal forwarder using a service account with full permission (777) and also have tried multiple release of Splunk (version 6.5.2 and 6.5.9 ) on an AIX Server 7.1 . But unable to start splunk on this server due the following errors .
bash-4.3$ /opt/splunkforwarder/bin/splunk start --accept-license
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
Did not find "disabled" setting of "kvstore" stanza in server bundle.
Splunk> Finding your faults, just like mom.
Checking prerequisites...
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
Checking mgmt port [8089]: exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
open
Checking kvstore port [8191]: exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
open
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
exec(): 0509-036 Cannot load program /opt/splunkforwarder/bin/splunkd because of the following errors:
0509-150 Dependent module /usr/lib/libcrypto.so could not be loaded.
0509-022 Cannot load module /usr/lib/libcrypto.so.
0509-026 System error: Cannot run a file that does not have a valid format.
SSL certificate generation failed.
Yes, check your path and packages:
0509-022 Cannot load module /usr/lib/libcrypto.so
tells you that the forwarder needs this library in the $LD_LIBRARY_PATH and if it cannot find it, it won't start. Btw: this is common on lots of *nixes, linux binaries have similar prerequisites... not only with Splunk binaries. I wonder why your libcrypto.so is missing. Either it's not on the system (try find) or your AIX Admin did not provide you the correct path to it.
Any customisations to your LIBPATH or LD_LIBRARY_PATH? This can trigger issues with finding the libraries within AIX...
You might want to
echo $LIBPATH
echo $LD_LIBRARY_PATH
To check