Issues with Splunk 8.1 Install

[MichaelMcAleer]

Hi Splunk Community,

Im trying to install Splunk on a new VM as part of funcitonal testing for an add-on, I would build these daily and havnt run into any issues before today with the process.

Running on Ubuntu 20.04.1, all packages up to date as of today. During the initial dpkg -i phase the following error is reported but install completes anyway:

cp: cannot stat '/opt/splunk/etc/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag': No such file or directory

I go through the process of starting splunk, accepting license, creating initial user, all fine, but when I get to the GUI and select 'browse more apps' I get the following error back:

 Error connecting: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.. Your Splunk instance is specifying custom CAs to trust using sslRootCAPath configuration in server.conf's [sslConfig] stanza. Make sure the CAs in the appsCA.pem (located under $SPLUNK_HOME/etc/auth/appsCA.pem) are included in the CAs specified by sslRootCAPath. To do this, append appsCA.pem to the file specified by the sslRootCAPath parameter.

Any idea what is going on here? I have another VM on the same host, identical network configuration, OS versions, its just been running for a few weeks. The Splunk env that is having issues is on a clean VM.

I have tried this using the same install .deb from the working VM and again with the latest .deb from the Splunk website, both exhibit same issue so im thinking it may be a system package dependency. 

Thanks!

Michael

[CryptoJones]

Had the same issue on fresh install of Ubuntu Server 20.04 LTS. Thank you for creating this post!

[MichaelMcAleer]

Never had an issue running splunk as root user or installing as root user. Given Splunk creates these users during install, as with the /opt/splunk directory, how can I set ownership for a folder/user that doesn't exist yet?

[jeffh-cf]

I am receiving the same error message when installing from:

splunk-8.1.0-f57c09e87251-linux-2.6-amd64.deb

I am using a fresh installation of Ubuntu 20.04 running on AWS from the AWS Marketplace image.

user@vmserver:~$ sudo dpkg -i splunk-8.1.0-f57c09e87251-linux-2.6-amd64.deb

Selecting previously unselected package splunk.

(Reading database ... 63367 files and directories currently installed.)

Preparing to unpack splunk-8.1.0-f57c09e87251-linux-2.6-amd64.deb ...

Unpacking splunk (8.1.0) ...

Setting up splunk (8.1.0) ...

cp: cannot stat '/opt/splunk/etc/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag': No such file or directory

complete

This is a Ubuntu Server VM, so there is no GUI for me to verify whether or not I get the same SSL error message.

[vishaltv]

++ Apologies, I missed to add a step ; reposting the same 
===========================================

I faced same issue while installing 8.1.1 in ubuntu. 

Setting up splunk (8.1.1) ...
cp: cannot stat '/opt/splunk/etc/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag': No such file or directory
complete

And we I checked, I could see a folder /opt/splunk/swidtag/  and a file "splunk-Splunk-Enterprise-primary.swidtag"

I copied it to "/opt/splunk/etc/"  and  renamed it to "regid.2001-12.com.splunk-Splunk-Enterprise.swidtag'" 

Change Owner of the file :

 /opt/splunk/etc# chown splunk:splunk regid.2001-12.com.splunk-Splunk-Enterprise.swidtag

Install again and it Worked!!

sudo dpkg -i splunk-8.1.1-08187535c166-linux-2.6-amd64.deb
(Reading database ... 265846 files and directories currently installed.)
Preparing to unpack splunk-8.1.1-08187535c166-linux-2.6-amd64.deb ...
This looks like an upgrade of an existing Splunk Server. Attempting to stop the installed Splunk Server...
splunkd is not running.
Unpacking splunk (8.1.1) over (8.1.1) ...
Setting up splunk (8.1.1) ...
complete

[subhrobera2021]

Accurate and it worked .

Great!!

[vishaltv]

I faced same issue while installing 8.1.1 in ubuntu. 

Setting up splunk (8.1.1) ...
cp: cannot stat '/opt/splunk/etc/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag': No such file or directory
complete

And we I checked, I could see a folder /opt/splunk/swidtag/  and a file "splunk-Splunk-Enterprise-primary.swidtag"

I copied it to "/opt/splunk/etc/"  and  renamed it to "regid.2001-12.com.splunk-Splunk-Enterprise.swidtag'" 

And it Worked!!

sudo dpkg -i splunk-8.1.1-08187535c166-linux-2.6-amd64.deb
(Reading database ... 265846 files and directories currently installed.)
Preparing to unpack splunk-8.1.1-08187535c166-linux-2.6-amd64.deb ...
This looks like an upgrade of an existing Splunk Server. Attempting to stop the installed Splunk Server...
splunkd is not running.
Unpacking splunk (8.1.1) over (8.1.1) ...
Setting up splunk (8.1.1) ...
complete


thanks to @ljonsson

[ljonsson · Accepted solution]

The problem seems to be with the regid.2001-12.com.splunk-Splunk-Enterprise.swidtag file and how it is handled during installation.

In the postinst script of the deb package (extract with dpkg -e splunk-8.1...deb, then look at DEBIAN/postinst) there is a function called copy_swid_tag() that boils down to:

SWID_SRC_DIR="$SPLUNK_HOME/etc"
SWID_DEST_DIR="/usr/share/regid.2001-12.com.splunk"
mkdir -p $SWID_DEST_DIR
cp "$SWID_SRC_DIR/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag" $SWID_DEST_DIR

Listing the contents of the latest 8.0 and 8.1 deb packages we see that the swidtag file that used to live in /opt/splunk/etc has been moved (and renamed) to /opt/splunk/swidtag

$ dpkg -c splunk-8.1.0.1-24fd52428b5a-linux-2.6-amd64.deb | grep '\.swidtag'
-rw-r--r-- root/root       599 2020-11-17 18:28 ./opt/splunk/swidtag/splunk-Splunk-Enterprise-primary.swidtag

$ dpkg -c splunk-8.0.7-cbe73339abca-linux-2.6-amd64.deb | grep '\.swidtag'
-rw-r--r-- root/root      1116 2020-10-23 08:08 ./opt/splunk/etc/regid.2001-12.com.splunk-Splunk-Enterprise.swidtag

The problem is that the swidtag file has been renamed, but the postinst script has not been updated to reflect that change.

Not sure what the swidtag file does, and perhaps it doesn't matter, but to complete the installation as the postinst script intended, copy the file in place by hand

$ sudo cp /opt/splunk/swidtag/splunk-Splunk-Enterprise-primary.swidtag /usr/share/regid.2001-12.com.splunk
$ sudo chown splunk:splunk /usr/share/regid.2001-12.com.splunk/splunk-Splunk-Enterprise-primary.swidtag

BTW, this is all true for rpm packages as well

$ rpm -ql -p splunk-8.1.0.1-24fd52428b5a-linux-2.6-x86_64.rpm | grep '\.swidtag'
/opt/splunk/swidtag/splunk-Splunk-Enterprise-primary.swidtag

$ rpm -q --scripts -p splunk-8.1.0.1-24fd52428b5a-linux-2.6-x86_64.rpm  | less

[kpkeimig]

same issue for linux rpm forwarders (8.1.4).  issued a support case

[CryptoJones]

This fixed my issue. Thank you soo much!

[MichaelMcAleer]

I've had a number of issues so far using vanilla Ubuntu & Splunk 8.x but so far they haven't impacted development so I can just continue on with limited functionality.

Thanks for the detailed breakdown and answer!

[ITWhisperer]

Check the ownership of the files under /opt/splunk. They should all be owned by the splunk user but you will probably find that some are owned by root. Change the ownership to splunk and also make sure you change the user that splunk runs as so that it doesn't create any more files owned by root.

[jeffh-cf]

This issue occurs while installing Splunk, not during the startup. It's before Splunk is even started for the first time.

I have verified that /opt/splunk and all subdirectories are owned by splunk:splunk

I'm wondering if the message benign?

[ITWhisperer]

I got the same message when installing on ubuntu 20.04 (on WSL, but that probably doesn't matter), and I found files owned by root and splunk processes running as root. When I resolved these (and the optimistic file lock required by ubuntu on WSL), the problem went away. If you don't have these then it is probably something different.

[jeffh-cf]

@ITWhisperer thanks for your help. Everything is working as expected.