Getting Data In

Splunk Universal Forwarder On AIX Fails To Start After Upgrade To Spunk 6.4.x

dshakespeare_sp
Splunk Employee
Splunk Employee

After upgrading Splunk Universal Forwarder to version 6.4.0 or above, Splunk will no longer start and the following error is reported.

Could not load program splunkd:
Symbol resolution failed for splunkd because:
Symbol SSL_is_server (number 648) is not exported from dependent
module /apps/splunkforwarder/lib/libssl.so.
Examine .loader section symbols with the 'dump -Tv' command.

How can I resolve this error?

dshakespeare_sp
Splunk Employee
Splunk Employee

This issue is caused by the fact some library files have failed to update. The standard AIX tar command will report certain files in $SPLUNK_HOME/lib as being in use when they are cached in library memory.

tar xvf ./splunkforwarder-6.4.1-debde650d26e-AIX-powerpc.tar
...
tar: 0511-188 Cannot create splunkforwarder/lib/libarchive.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libbz2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libcrypto.so.1.0.0: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libexslt.a, 452512 bytes, 884 media blocks.
tar: 0511-188 Cannot create splunkforwarder/lib/libpcre.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libsqlite3.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libssl.so.1.0.0: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxml2.a: Cannot open or remove a file containing a running program.
tar: 0511-188 Cannot create splunkforwarder/lib/libxslt.a: Cannot open or remove a file containing a running program.
x splunkforwarder/lib/libz.a, 1353663 bytes, 2644 media blocks.

To resolve this issue;

EITHER
Run the upgrade again using GNU tar. We always used to recommend this method for Splunk Enterprise installs on AIX (see http://docs.splunk.com/Documentation/Splunk/6.2.11/Installation/InstallonAIX)

GNU tar is typically installed as part of the AIX Toolbox for Linux Applications package included in the base AIX install. It is located in /opt/freeware/bin/tar

OR
1. Run the AIX slibclean command (see man slibclean) which will attempt to remove any currently unused modules in kernel and library memory
2. Re run the upgrade procedure.

jrodman
Splunk Employee
Splunk Employee

post cleanup, you might want to try 'splunk validate files' to be sure the files on disk now match the files in the provided manifest.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...