Splunk Universal Forwarder On AIX Fails To Start After Upgrade To Spunk 6.4.x

[dshakespeare_sp]

After upgrading Splunk Universal Forwarder to version 6.4.0 or above, Splunk will no longer start and the following error is reported.

Could not load program splunkd:
Symbol resolution failed for splunkd because:
Symbol SSL_is_server (number 648) is not exported from dependent
module /apps/splunkforwarder/lib/libssl.so.
Examine .loader section symbols with the 'dump -Tv' command.

How can I resolve this error?

[dshakespeare_sp]

This issue is caused by the fact some library files have failed to update. The standard AIX tar command will report certain files in $SPLUNK_HOME/lib as being in use when they are cached in library memory.

tar xvf ./splunkforwarder-6.4.1-debde650d26e-AIX-powerpc.tar
...
tar: 0511-188 Cannot create splunkforwarder/lib/libarchive.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libbz2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libcrypto.so.1.0.0: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libexslt.a, 452512 bytes, 884 media blocks.
tar: 0511-188 Cannot create splunkforwarder/lib/libpcre.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libsqlite3.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libssl.so.1.0.0: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxml2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxslt.a: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libz.a, 1353663 bytes, 2644 media blocks.

To resolve this issue;

EITHER
Run the upgrade again using GNU tar. We always used to recommend this method for Splunk Enterprise installs on AIX (see http://docs.splunk.com/Documentation/Splunk/6.2.11/Installation/InstallonAIX)

GNU tar is typically installed as part of the AIX Toolbox for Linux Applications package included in the base AIX install. It is located in /opt/freeware/bin/tar

OR
1. Run the AIX slibclean command (see man slibclean) which will attempt to remove any currently unused modules in kernel and library memory
2. Re run the upgrade procedure.

[jrodman]

post cleanup, you might want to try 'splunk validate files' to be sure the files on disk now match the files in the provided manifest.