Getting Data In

Splunk Universal Forwarder On AIX Fails To Start After Upgrade To Spunk 6.4.x

dshakespeare_sp
Splunk Employee
Splunk Employee

After upgrading Splunk Universal Forwarder to version 6.4.0 or above, Splunk will no longer start and the following error is reported.

Could not load program splunkd:
Symbol resolution failed for splunkd because:
Symbol SSL_is_server (number 648) is not exported from dependent
module /apps/splunkforwarder/lib/libssl.so.
Examine .loader section symbols with the 'dump -Tv' command.

How can I resolve this error?

dshakespeare_sp
Splunk Employee
Splunk Employee

This issue is caused by the fact some library files have failed to update. The standard AIX tar command will report certain files in $SPLUNK_HOME/lib as being in use when they are cached in library memory.

tar xvf ./splunkforwarder-6.4.1-debde650d26e-AIX-powerpc.tar
...
tar: 0511-188 Cannot create splunkforwarder/lib/libarchive.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libbz2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libcrypto.so.1.0.0: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libexslt.a, 452512 bytes, 884 media blocks.
tar: 0511-188 Cannot create splunkforwarder/lib/libpcre.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libsqlite3.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libssl.so.1.0.0: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxml2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxslt.a: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libz.a, 1353663 bytes, 2644 media blocks.

To resolve this issue;

EITHER
Run the upgrade again using GNU tar. We always used to recommend this method for Splunk Enterprise installs on AIX (see http://docs.splunk.com/Documentation/Splunk/6.2.11/Installation/InstallonAIX)

GNU tar is typically installed as part of the AIX Toolbox for Linux Applications package included in the base AIX install. It is located in /opt/freeware/bin/tar

OR
1. Run the AIX slibclean command (see man slibclean) which will attempt to remove any currently unused modules in kernel and library memory
2. Re run the upgrade procedure.

jrodman
Splunk Employee
Splunk Employee

post cleanup, you might want to try 'splunk validate files' to be sure the files on disk now match the files in the provided manifest.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...