topic Re: Universal Forwarder Problem in Installation

Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 13:41:41 GMT

Here is my situation, we had a test installation of Splunk Enterprise 7.2.4 on Server1, with Universal Forwarders installed on Server2 & Server3, just generic Windows Servers. Events were arriving in Splunk Enterprise, and all was good until our license expired.

We then received a Developer License so that my Federal Agency can test it.

I uninstalled the Universal Forwarder via Add/Remove Programs & rebooted.

Then since 7.2.5 was released I downloaded that and installed Splunk Enterprise on new Server4 and installed the Universal Forwarders (7.2.5) on Server2 & Server3.

I am not getting events into Splunk Enterprise, so I thought I would uninstall & reinstall the Universal Forwarder on Server2 & Server3 again.

It will not install, the old Universal Forwarder folders and files still exist, but I cannot either Install, Repair, or Uninstall now.

What do I do to remove the old UF installation and reinstall?

Re: Universal Forwarder Problem

nickhills — Wed, 20 Mar 2019 14:27:04 GMT

Have you looked in the %TEMP%/splunkInstall.log file - If the install is failing it should contain details on the reason why.

Re: Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 14:33:13 GMT

This is a Snippet:

9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

Re: Universal Forwarder Problem

nickhills — Wed, 20 Mar 2019 14:55:04 GMT

The last 3 lines:

Removing service SplunkForwarder
Service removed
Disabled.

Suggest that the uninstall completed. What is written to that log file if you reinstall again?

Re: Universal Forwarder Problem

carbin — Tue, 29 Sep 2020 23:46:22 GMT

cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T09:20:54-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.
11:12:37 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" _internal first-time-run --answer-yes --no-prompt >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=7.2.5
BUILD=088f49762779
PRODUCT=splunk
PLATFORM=Windows-AMD64

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.

"N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem": already a renewed Splunk certificate: skipping renewal
"N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem": already a renewed Splunk certificate: skipping renewal
[App Key Value Store migration] Binary for service(34) is missing.
[App Key Value Store migration] Binary for service(34) is missing.

-- Migration information is being logged to 'N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.11-12-38' --
11:12:41 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" _internal pre-flight-checks --answer-yes --no-prompt >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'N:\Program Files\SplunkUniversalForwarder\splunkforwarder-7.2.5-088f49762779-windows-64-manifest'
All installed files intact.
Done
11:12:45 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd install --startup=auto >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Installing service SplunkForwarder
Service installed
11:12:46 AM
cmd.exe /c "icacls "N:\Program Files\SplunkUniversalForwarder\etc" /T /C /grant *S-1-5-32-544:f >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
processed file: N:\Program Files\SplunkUniversalForwarder\etc
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth
processed file: N:\Program Files\SplunkUniversalForwarder\etc\copyright.txt
processed file: N:\Program Files\SplunkUniversalForwarder\etc\datetime.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\deployment-apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\disabled-apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-btool-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-btool.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-cmdline-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-cmdline.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-utility.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall
processed file: N:\Program Files\SplunkUniversalForwarder\etc\passwd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\prettyprint.xsl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk.version
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system
processed file: N:\Program Files\SplunkUniversalForwarder\etc\users
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\bin
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\bin\collector.path
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\restmap.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\transforms.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\default-mode.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\limits.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\web.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata\local.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth.rnd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\appsCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\appsLicenseCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.srl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cloudCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\crl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\splunk.secret
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\crl\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release\ca.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release\cacert.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\deployment-apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\disabled-apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses\forwarder
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses\forwarder\splunkforwarder.lic
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\parsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\exec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\fschangemanager
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\RemoteQueue
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\structuredparsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\tailfile
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\TCP
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\UDP
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\wineventlog
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\winparsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\exec\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\fschangemanager\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\RemoteQueue\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\structuredparsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\tailfile\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\TCP\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\UDP\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\wineventlog\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\winparsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\parsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml.cfg-default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\users
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\users\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\admon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\MonitorNoHandle.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\perfmon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\powershell.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\powershell2.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\winEventLog.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinHostMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinNetMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinPrintMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinRegMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\alert_actions.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\audit.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\authentication.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\authorize.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\conf.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\default-mode.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\health.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\limits.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\literals.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\livetail.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\messages.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\procmon-filters.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\restmap.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\source-classifier.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\sourcetypes.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\telemetry.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\visualizations.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\web.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\workload_pools.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\workload_rules.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\authentication.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\migration.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\user-seed.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata\local.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\alert_actions.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\alert_actions.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\audit.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\audit.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authentication.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authentication.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authorize.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authorize.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\checklist.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\collections.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\collections.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\conf_checker.rules
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default-mode.conf.examples
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default-mode.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default.meta.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default.meta.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\deploymentclient.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\deploymentclient.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\health.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\health.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\inputs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\inputs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\instance.cfg.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\instance.cfg.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\limits.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\limits.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\literals.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\literals.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\livetail.conf.examples
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\livetail.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\messages.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\messages.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\migration.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\outputs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\outputs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\passwords.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\passwords.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\procmon-filters.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\procmon-filters.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\props.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\props.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\restmap.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\restmap.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\server.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\server.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\serverclass.seed.xml.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\source-classifier.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\source-classifier.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\sourcetypes.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\sourcetypes.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\splunk-launch.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-prefs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-prefs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-seed.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-seed.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\visualizations.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\web.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\web.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\wmi.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\wmi.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_pools.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_pools.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_rules.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_rules.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static\atom.xsl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static\splunkrc_cmds.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\users\users.ini
Successfully processed 230 files; Failed processing 0 files
11:12:46 AM
cmd.exe /c "icacls "N:\Program Files\SplunkUniversalForwarder\var" /T /C /grant *S-1-5-32-544:f >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
processed file: N:\Program Files\SplunkUniversalForwarder\var
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib
processed file: N:\Program Files\SplunkUniversalForwarder\var\log
processed file: N:\Program Files\SplunkUniversalForwarder\var\run
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\authDb
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\hashDb
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinputs
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\rawdata
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinputs\WinEventLog
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\audit
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\introspection
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\watchdog
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\first_install.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.09-01-52
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.09-20-44
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.11-12-38
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\run.rnd
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\cachemanager_upload.json
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\upload
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\i18n
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules\static
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules\static\css
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool\dirmoncache
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool\splunk
Successfully processed 35 files; Failed processing 0 files
11:12:46 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 15:12:48 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T11:12:48-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:12:48 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 15:12:48 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

Re: Universal Forwarder Problem

nickhills — Wed, 20 Mar 2019 15:50:00 GMT

If you go to add-remove programs is the Splunk UF now gone?
If not, remove it from add/remove etc
Then move the folder from program files, and reinstall again.
It seems maybe the old installation directory is causing some odd behavior.

Re: Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 15:58:56 GMT

So, I moved the entire SplunkUniversalForwarder folder from Program Files to a temp folder and tried again:

…
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 15:56:25 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T11:56:25-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:56:25 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 201 Created
Date: Wed, 20 Mar 2019 15:56:25 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 4439
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server
2019-03-20T11:56:25-04:00

<name>Splunk</name>

1
30
0

<title>165.112.254.26:9997</title>
<id>/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997</id>
<updated>1969-12-31T19:00:00-05:00</updated>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="alternate"/>
<author>
  <name>nobody</name>
</author>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="list"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/_reload" rel="_reload"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="edit"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="remove"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/allconnections" rel="allconnections"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/disable" rel="disable"/>
<content type="text/xml">
  <s:dict>
    <s:key name="destHost">165.112.254.26</s:key>
    <s:key name="destIp">165.112.254.26</s:key>
    <s:key name="destPort">9997</s:key>
    <s:key name="eai:acl">
      <s:dict>
        <s:key name="app">system</s:key>
        <s:key name="can_change_perms">1</s:key>
        <s:key name="can_list">1</s:key>
        <s:key name="can_share_app">1</s:key>
        <s:key name="can_share_global">1</s:key>
        <s:key name="can_share_user">0</s:key>
        <s:key name="can_write">1</s:key>
        <s:key name="modifiable">1</s:key>
        <s:key name="owner">nobody</s:key>
        <s:key name="perms">
          <s:dict>
            <s:key name="read">
              <s:list>
                <s:item>*</s:item>
              </s:list>
            </s:key>
            <s:key name="write">
              <s:list>
                <s:item>*</s:item>
              </s:list>
            </s:key>
          </s:dict>
        </s:key>
        <s:key name="removable">1</s:key>
        <s:key name="sharing">system</s:key>
      </s:dict>
    </s:key>
    <s:key name="method">autobalance</s:key>
    <s:key name="sourcePort">8089</s:key>
    <s:key name="status">not_connected</s:key>
  </s:dict>
</content>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:56:26 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.

Re: Universal Forwarder Problem

nickhills — Wed, 20 Mar 2019 16:16:15 GMT

Is 165.112.254.26 the IP of the old server, or the new one?

I had been assuming that it was the old server?

If you run a search on your new Splunk server for:
index=_internal |stats count by host
Do you see your forwarders in the results?

Re: Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 16:20:32 GMT

That is the IP of the new server. When I run that search, it only returns one Host, the Splunk Ent server. I'm thinking Registry Keys must be holding me up

Re: Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 17:27:54 GMT

In desperation I went to my other test server and did the following:

1: Verified that Add/Remove Programs did NOT contain Universal Forwarder
2: Moved the original install folder to a new Temp folder
3: Opened Regedit and searched for "splunk"
4: Deleted all that matched
5: Rebooted server
6: Attempted Installation. It failed and rolled back.

Did I miss Registry Keys?

Re: Universal Forwarder Problem

carbin — Wed, 20 Mar 2019 17:27:54 GMT

In desperation I went to my other test server and did the following:

Did I miss Registry Keys?

Re: Universal Forwarder Problem

nickhills — Thu, 21 Mar 2019 10:50:40 GMT

You can try this, which I found referenced in another post.
It will forcefully remove a UF, but please read the warnings, and satisfy yourself that you are comfortable before executing it.

https://github.com/dstaulcu/SplunkTools/blob/master/Remove-UniversalForwarder-BrokenMSI.ps1

Re: Universal Forwarder Problem

carbin — Thu, 21 Mar 2019 12:19:38 GMT

Thanks for the suggestion. I executed that PS script as admin, rebooted and still won't install. I may have to see if I can revert to an earlier snapshot of the VM (if one exists), but I'm unsure what I would do differently. Will the 7.2.5 installed upgrade or install over a 7.2.4 install?