Why is a silent installation of a universal forwarder failing with "CopyCerts: Error 0x80004005: Cannot copy certificates."?

[123BLiN]

Hello I'm trying to prepare script for silent install of UniversalForwarder to automate installing it on many hosts.
Here is arguments list:

/i splunkforwarder-6.2.2-255606-x64-release.msi CERTFILE='C:\Users\Administrator\Desktop\splunk-forwarder\preparedServer
Certificate.pem' ROOTCACERTFILE='C:\Users\Administrator\Desktop\splunk-forwarder\rootCACertificate.pem' CERTPASSWORD='pa
ssForKey' DEPLOYMENT_SERVER='splunk:8089' RECEIVING_INDEXER='splunk:9997' WINEVENTLOG_SEC_ENABLE=1 WINEVENTLOG_SYS_ENABL
E=1 WINEVENTLOG_APP_ENABLE=1 PERFMON=cpu,memory AGREETOLICENSE=Yes /quiet /norestart /log install-log.txt

This is clean Amazon last AMI Windows 2012R2 installation x64
Here what I see in log file install-log.txt:

Action start 20:35:47: InstallFinalize.
InstallRegmonDrv:  Warning: Invalid property ignored: FailCA=.
InstallRegmonDrv:  Info: Driver inf file: C:\Program Files\SplunkUniversalForwarder\bin\splunkdrv-win6.inf.
InstallNetmonDrv:  Warning: Invalid property ignored: FailCA=.
InstallNetmonDrv:  Info: Driver inf file: C:\Program Files\SplunkUniversalForwarder\bin\splknetdrv-vista.inf.
InstallNohandleDrv:  Warning: Invalid property ignored: FailCA=.
InstallNohandleDrv:  Info: Driver inf file: C:\Program Files\SplunkUniversalForwarder\bin\SplunkMonitorNoHandleDrv-vista.inf.
CopyCerts:  Warning: Invalid property ignored: FailCA=.
CopyCerts:  Info: Certificate file: 'C:\Users\Administrator\Desktop\splunk-forwarder\preparedServerCertificate.pem', root certificate file: 'C:\Users\Administrator\Desktop\splunk-forwarder\rootCACertificate.pem'.
CopyCerts:  Error: Cannot copy certificate file: 'C:\Users\Administrator\Desktop\splunk-forwarder\preparedServerCertificate.pem'.
CopyCerts:  Error 0x80004005: Cannot copy certificates.
CustomAction CopyCerts returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 20:35:52: InstallFinalize. Return value 3.

Will be gratefull for any help!
Second question is:

• could I use somehow some key during silent install to told client check server sertificate (sslVerifyServerCert=true)

Thanks in advance!

[matthewjohnson]

In my case, the problem was related to permissions on the source. I suspect the copy operation runs under some non-privileged token. The solution in my case was to copy the certificates using a batch script from the source (a network path) to a temporary folder in the install path. The install processed copied the certificates again from my temporary location to the /etc/auth folder.

Here's my batch file if it helps.

@echo off
setlocal enableextensions

if "%1" == "" goto NoPassword

set SPLUNK_HOME=%PROGRAMFILES%\SplunkUniversalForwarder
set SPLUNK_CERTS_TEMP=%SPLUNK_HOME%\Certs_Temp
set CERT_CA_SOURCE=%SPLUNK_CERTS_TEMP%\MYRootCA.pem
set CERT_CLIENT_SOURCE=%SPLUNK_CERTS_TEMP%\splunkuf.full.pem

echo mkdir "%SPLUNK_CERTS_TEMP%"
echo ---------------------------------
mkdir "%SPLUNK_CERTS_TEMP%"
@echo. 
@echo.

echo copy "%~dp0*.pem" "%SPLUNK_CERTS_TEMP%"
echo ---------------------------------
copy "%~dp0*.pem" "%SPLUNK_CERTS_TEMP%"
@echo. 
@echo. 

echo msiexec.exe /i "%~dp0splunkforwarder-6.3.2-aaff59bb082c-x64-release.msi" AGREETOLICENSE=Yes RECEIVING_INDEXER=splunk.place:9997 DEPLOYMENT_SERVER=splunk.place:8089 LAUNCHSPLUNK=0 ROOTCACERTFILE="%CERT_CA_SOURCE%" CERTFILE="%CERT_CLIENT_SOURCE%" CERTPASSWORD=%1 /quiet /L*v! "%SPLUNK_HOME%\msiexec_install.log"
echo ---------------------------------
msiexec.exe /i "%~dp0splunkforwarder-6.3.2-aaff59bb082c-x64-release.msi" AGREETOLICENSE=Yes RECEIVING_INDEXER=splunk.place:9997 DEPLOYMENT_SERVER=splunk.place:8089 LAUNCHSPLUNK=0 ROOTCACERTFILE="%CERT_CA_SOURCE%" CERTFILE="%CERT_CLIENT_SOURCE%" CERTPASSWORD=%1 /quiet /L*v! "%SPLUNK_HOME%\msiexec_install.log"
@echo. 
@echo. 


echo rmdir /S /Q "%SPLUNK_CERTS_TEMP%"
echo ---------------------------------
rmdir /S /Q "%SPLUNK_CERTS_TEMP%"

goto End

:NoPassword

echo You must specify the certificate password at the command line:
echo Example: deploy.bat <password>

:End

pause

Unfortunately, once I managed to work around the error, my next problems was that no SSL configuration data was added to the outputs.conf file (although my certificates were copied to the /etc/auth folder). Frustrating! I briefly explored using command line options after install to configure outputs.conf, but not all settings can be configured at the command line - compressed = true, for example.

Ultimately, I decided to skip configuring Outputs.conf via install parameters, and instead created a Splunk app with basic configuration data and used forwarder management to deploy it. The client certificate is included in the app, which will make it easy to update in the future. I'm using the same client certificate for groups of universal forwarders, which simplifies my deployment. If you're using a unique certificate for each UF, things might not be so straightforward...

My basic process was:

Create: $SPLUNK_HOME/etc/deployment-apps/Base_Win_Server_UF_Config/certs to contains my UF client certificates
Create: $SPLUNK_HOME/etc/deployment-apps/Base_Win_Server_UF_Config/local/outputs.conf (should probably be .../default/outputs.conf instead)

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk.place:9997
compressed = true
sslRootCAPath = $SPLUNK_HOME/etc/apps/Base_Win_Server_UF_Config/certs/MYRootCA.pem
sslCertPath = $SPLUNK_HOME/etc/apps/Base_Win_Server_UF_Config/certs/splunkuf.full.pem
sslPassword = <Gets hashed on service start>
sslVerifyServerCert = true

[tcpout-server://splunk.place:9997]

Finally, I configured the app (restart on deploy) and server classes in Splunk. Works like a charm so far!

Hopefully, this information can same someone a few hours of frustration in the future.