How to install Splunk forwarder with powershell?

[JacksonModlin]

Hello all,

We are starting to integrate spunk into our systems, and in order to make sure everything goes smoothly we want to write a PowerShell script for the installation.

We use Splunk Cloud, so we are unsure if there is a way to set a PowerShell script to install it across our systems. We would like guidance where possible.

Thank you

[richgalloway]

This answer should help: https://community.splunk.com/t5/Getting-Data-In/Powershell-unattended-installation/m-p/81069

The installation instructions at https://docs.splunk.com/Documentation/Forwarder/9.0.1/Forwarder/InstallaWindowsuniversalforwarderfro... should help you understand what the script is doing.

[JacksonModlin]

I saw many posts like this one, looking at install the forwarder using PowerShell, but we need to install it for splunk cloud, will it still work, or will we have to install everything manually?

[richgalloway]

Forwarders are installed the same way for Splunk Cloud as for Splunk Enterprise.

For Splunk Cloud, there is an additional step to install the Splunk Cloud credentials.  Get the creds by going to the Universal Forwarder app on your Splunk Cloud search head and clicking on the green Download button.  The downloaded file should be expanded into the SplunkUniversalForwarder\etc\apps folder.

[JacksonModlin]

We tried this, and it failed. It keeps trying to install it as a splunk enterprise instance instead of splunk cloud, is there a flag we can use to make the version of the forwarder into a splunk cloud instance????

Attached is our log file, please provide guidance whenever possible. 

```

10-31-2022 14:44:12.078 -0500 INFO LMStackMgr [0 MainThread] - Initializing CleMgr...
10-31-2022 14:44:12.079 -0500 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - serverName=IT-JMODLIN guid=96BF5E48-20C5-4825-AA20-94D331BD980E
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - connection_timeout=30
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - send_timeout=30
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - receive_timeout=30
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - squash_threshold=2000
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - strict_pool_quota=1
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
10-31-2022 14:44:12.079 -0500 INFO LMConfig [0 MainThread] - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
10-31-2022 14:44:12.079 -0500 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr runContext_splunkd=false
10-31-2022 14:44:12.079 -0500 INFO LMStackMgr [0 MainThread] - closing stack mgr
10-31-2022 14:44:12.079 -0500 INFO LMSlaveInfo [0 MainThread] - all slaves cleared
10-31-2022 14:44:12.079 -0500 INFO LMStackMgr [0 MainThread] - Initalized license_warnings_update_interval=auto
10-31-2022 14:44:12.079 -0500 INFO LMStackMgr [0 MainThread] - License Manager supports Conditional Licensing Enforcement. For baked in CLE policies, window_period=60 days, max_violations=45, for stack size below 107374182400 bytes
10-31-2022 14:44:12.079 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for free
10-31-2022 14:44:12.079 -0500 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=3 MaxSize=0 isDefault=1 features= for free
10-31-2022 14:44:12.080 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
10-31-2022 14:44:12.080 -0500 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
10-31-2022 14:44:12.081 -0500 INFO LMStack [0 MainThread] - Added type=forwarder license, from file=splunkforwarder.lic, to stack=forwarder of group=Forwarder
10-31-2022 14:44:12.081 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
10-31-2022 14:44:12.081 -0500 INFO LMStackMgr [0 MainThread] - created stack='forwarder'
10-31-2022 14:44:12.081 -0500 INFO LMStackMgr [0 MainThread] - Replaced with latest policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
10-31-2022 14:44:12.081 -0500 INFO LMStackMgr [0 MainThread] - Initialized hideQuotaWarning = "0"
10-31-2022 14:44:12.081 -0500 INFO LMStackMgr [0 MainThread] - init completed [96BF5E48-20C5-4825-AA20-94D331BD980E,Forwarder,runContext_splunkd=false]
10-31-2022 14:44:12.081 -0500 INFO LicenseMgr [0 MainThread] - StackMgr init complete...
10-31-2022 14:44:12.081 -0500 INFO LMTracker [0 MainThread] - Setting default product type='enterprise'
10-31-2022 14:44:12.081 -0500 INFO LMTracker [0 MainThread] - this is not splunkd, will perform partial init
10-31-2022 14:44:12.081 -0500 INFO LicenseMgr [0 MainThread] - Tracker init complete...
10-31-2022 14:44:12.086 -0500 INFO KVStorageEngineUpgrade [0 MainThread] - Setting parallelDumpCollectionJobs=0 parallelRestoreCollectionJobs=0 based on max_num_cpus=8 and total_configured_collections=0, insertionWorkersPerCollection=1 as maxInsertionWorkersPerCollection=4
10-31-2022 14:44:12.090 -0500 INFO KVStoreBackupRestore [0 MainThread] - Before KV Store engine migration: KVStoreDbsize=4096 fsBytesFree=90732716032
10-31-2022 14:44:12.090 -0500 WARN SSLOptions [17672 MainThread] - server.conf/[kvstore]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Starting mongod with executable name=mongod-4.0.exe version=kvstore version 4.0
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --dbpath C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\kvstore\mongo
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --storageEngine wiredTiger
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using cacheSize=2.25GB
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --port 8191
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --timeStampFormat iso8601-utc
10-31-2022 14:44:12.094 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --oplogSize 200
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --keyFile C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\kvstore\mongo\splunk.key
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Starting mongod in standalone mode
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses)
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslMode requireSSL
10-31-2022 14:44:12.095 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslAllowInvalidHostnames
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Found an existing PFX certificate
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Found an existing PFX certificate
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslCertificateSelector subject=SplunkServerDefaultCert
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslAllowInvalidCertificates
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslAllowConnectionsWithoutCertificates
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslDisabledProtocols noTLS1_0,noTLS1_1
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
10-31-2022 14:44:12.321 -0500 INFO MongodRunner [17672 MainThread] - Using mongod command line --noscripting
10-31-2022 14:44:12.326 -0500 ERROR MongodRunner [17672 MainThread] - Failed to start mongod.
10-31-2022 14:59:13.939 -0500 ERROR KVStoreConfigurationProvider [17672 MainThread] - Cannot dump kvstore data reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=901610ms
10-31-2022 14:59:13.939 -0500 ERROR KVStoreConfigurationProvider [17672 MainThread] - Failed to receive response from kvstore error=, service not ready after waiting for timeout=901610ms
10-31-2022 15:01:27.887 -0500 INFO LMStackMgr [0 MainThread] - Initializing CleMgr...
10-31-2022 15:01:27.888 -0500 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - serverName=IT-JMODLIN guid=DBE7F1CA-C25F-430E-AB8C-DA68B1792CC4
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - connection_timeout=30
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - send_timeout=30
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - receive_timeout=30
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - squash_threshold=2000
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - strict_pool_quota=1
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
10-31-2022 15:01:27.888 -0500 INFO LMConfig [0 MainThread] - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
10-31-2022 15:01:27.888 -0500 INFO LicenseMgr [0 MainThread] - Initing LicenseMgr runContext_splunkd=false
10-31-2022 15:01:27.888 -0500 INFO LMStackMgr [0 MainThread] - closing stack mgr
10-31-2022 15:01:27.888 -0500 INFO LMSlaveInfo [0 MainThread] - all slaves cleared
10-31-2022 15:01:27.889 -0500 INFO LMStackMgr [0 MainThread] - Initalized license_warnings_update_interval=auto
10-31-2022 15:01:27.889 -0500 INFO LMStackMgr [0 MainThread] - License Manager supports Conditional Licensing Enforcement. For baked in CLE policies, window_period=60 days, max_violations=45, for stack size below 107374182400 bytes
10-31-2022 15:01:27.889 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for free
10-31-2022 15:01:27.889 -0500 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=3 MaxSize=0 isDefault=1 features= for free
10-31-2022 15:01:27.889 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
10-31-2022 15:01:27.889 -0500 INFO LMStackMgr [0 MainThread] - Added policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
10-31-2022 15:01:27.891 -0500 INFO LMStack [0 MainThread] - Added type=forwarder license, from file=splunkforwarder.lic, to stack=forwarder of group=Forwarder
10-31-2022 15:01:27.891 -0500 INFO LMLicense [0 MainThread] - Applying default enforcement policy for forwarder
10-31-2022 15:01:27.891 -0500 INFO LMStackMgr [0 MainThread] - created stack='forwarder'
10-31-2022 15:01:27.891 -0500 INFO LMStackMgr [0 MainThread] - Replaced with latest policy WinSz=30 Warnings=5 MaxSize=0 isDefault=1 features= for forwarder
10-31-2022 15:01:27.891 -0500 INFO LMStackMgr [0 MainThread] - Initialized hideQuotaWarning = "0"
10-31-2022 15:01:27.891 -0500 INFO LMStackMgr [0 MainThread] - init completed [DBE7F1CA-C25F-430E-AB8C-DA68B1792CC4,Forwarder,runContext_splunkd=false]
10-31-2022 15:01:27.891 -0500 INFO LicenseMgr [0 MainThread] - StackMgr init complete...
10-31-2022 15:01:27.891 -0500 INFO LMTracker [0 MainThread] - Setting default product type='enterprise'
10-31-2022 15:01:27.892 -0500 INFO LMTracker [0 MainThread] - this is not splunkd, will perform partial init
10-31-2022 15:01:27.892 -0500 INFO LicenseMgr [0 MainThread] - Tracker init complete...
10-31-2022 15:01:27.899 -0500 INFO KVStorageEngineUpgrade [0 MainThread] - Setting parallelDumpCollectionJobs=0 parallelRestoreCollectionJobs=0 based on max_num_cpus=8 and total_configured_collections=0, insertionWorkersPerCollection=1 as maxInsertionWorkersPerCollection=4
10-31-2022 15:01:27.906 -0500 INFO KVStoreBackupRestore [0 MainThread] - Before KV Store engine migration: KVStoreDbsize=4096 fsBytesFree=90669641728
10-31-2022 15:01:27.907 -0500 WARN SSLOptions [17024 MainThread] - server.conf/[kvstore]/sslVerifyServerCert is false disabling certificate validation; must be set to "true" for increased security
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Starting mongod with executable name=mongod.exe version=kvstore version 4.2
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --dbpath C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\kvstore\mongo
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --storageEngine wiredTiger
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using cacheSize=2.25GB
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --port 8191
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --timeStampFormat iso8601-utc
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --oplogSize 200
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --keyFile C:\Program Files\SplunkUniversalForwarder\var\lib\splunk\kvstore\mongo\splunk.key
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Starting mongod in standalone mode
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses)
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslMode requireSSL
10-31-2022 15:01:27.911 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslAllowInvalidHostnames
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Found an existing PFX certificate
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Found an existing PFX certificate
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslCertificateSelector subject=SplunkServerDefaultCert
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslAllowInvalidCertificates
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslAllowConnectionsWithoutCertificates
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
10-31-2022 15:01:28.152 -0500 INFO MongodRunner [17024 MainThread] - Using mongod command line --noscripting
10-31-2022 15:01:28.158 -0500 ERROR MongodRunner [17024 MainThread] - Failed to start mongod.
10-31-2022 15:16:29.657 -0500 ERROR KVStoreConfigurationProvider [17024 MainThread] - Cannot dump kvstore data reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=901500ms
10-31-2022 15:16:29.657 -0500 ERROR KVStoreConfigurationProvider [17024 MainThread] - Failed to receive response from kvstore error=, service not ready after waiting for timeout=901500ms

```

[richgalloway]

I think I wasn't clear earlier.  The forwarder for Splunk Cloud is *exactly* the same as the one for Splunk Enterprise.  The destination of the data is the only difference.

It looks like you're running into a problem I've seen others report.  The universal forwarder doesn't use KVstore and should not be trying to install mongdb.  Try installing an earlier version of the forwarder to get around this until a fix comes out.

[JacksonModlin]

I downloaded version 9.0.0.1, and 8.2.8 and neither worked for our purposes, as both still resulted in mongDB still installing, is there any other guidance, you can provide, or is this the final verdict?

[richgalloway]

You can try disabling the KVstore by adding these lines to $SPLUNK_HOME/etc/system/local/server.conf

[kvstore]
disabled=true

[JacksonModlin]

I try that, and it still fails to install, i have started to log the file for installing, and the error is below, i have censored out sensitive data.

MSI (s) (98:C0) [09:20:28:480]: Note: 1: 1708
MSI (s) (98:C0) [09:20:28:480]: Note: 1: 2205 2: 3: Error
MSI (s) (98:C0) [09:20:28:480]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1708
MSI (s) (98:C0) [09:20:28:480]: Note: 1: 2205 2: 3: Error
MSI (s) (98:C0) [09:20:28:480]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (s) (98:C0) [09:20:28:480]: Product: UniversalForwarder -- Installation failed.

MSI (s) (98:C0) [09:20:28:481]: Windows Installer installed the product. Product Name: UniversalForwarder. Product Version: 9.0.1.0. Product Language: 1033. Manufacturer: Splunk, Inc.. Installation success or error status: 1603.

MSI (s) (98:C0) [09:20:28:494]: Deferring clean up of packages/files, if any exist
MSI (s) (98:C0) [09:20:28:494]: MainEngineThread is returning 1603
MSI (s) (98:00) [09:20:28:494]: No System Restore sequence number for this installation.
=== Logging stopped: 11/1/2022 9:20:28 ===
MSI (s) (98:00) [09:20:28:496]: User policy value 'DisableRollback' is 0
MSI (s) (98:00) [09:20:28:496]: Machine policy value 'DisableRollback' is 0
MSI (s) (98:00) [09:20:28:496]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (98:00) [09:20:28:496]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (98:00) [09:20:28:497]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (98:00) [09:20:28:497]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1
MSI (s) (98:00) [09:20:28:497]: Destroying RemoteAPI object.
MSI (s) (98:14) [09:20:28:497]: Custom Action Manager thread ending.
MSI (c) (08:4C) [09:20:28:498]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1
MSI (c) (08:4C) [09:20:28:500]: MainEngineThread is returning 1603
=== Verbose logging stopped: 11/1/2022 9:20:28 ===

[richgalloway]

I suggest requesting help from Splunk support if you can.