Installation
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

Universal Forwarder Problem

carbin
New Member
‎03-20-2019 06:41 AM

Here is my situation, we had a test installation of Splunk Enterprise 7.2.4 on Server1, with Universal Forwarders installed on Server2 & Server3, just generic Windows Servers. Events were arriving in Splunk Enterprise, and all was good until our license expired.

We then received a Developer License so that my Federal Agency can test it.

I uninstalled the Universal Forwarder via Add/Remove Programs & rebooted.

Then since 7.2.5 was released I downloaded that and installed Splunk Enterprise on new Server4 and installed the Universal Forwarders (7.2.5) on Server2 & Server3.

I am not getting events into Splunk Enterprise, so I thought I would uninstall & reinstall the Universal Forwarder on Server2 & Server3 again.

It will not install, the old Universal Forwarder folders and files still exist, but I cannot either Install, Repair, or Uninstall now.

What do I do to remove the old UF installation and reinstall?

Tags (1)
0 Karma
Reply

nickhills
Ultra Champion
‎03-21-2019 03:50 AM

You can try this, which I found referenced in another post.
It will forcefully remove a UF, but please read the warnings, and satisfy yourself that you are comfortable before executing it.

https://github.com/dstaulcu/SplunkTools/blob/master/Remove-UniversalForwarder-BrokenMSI.ps1

If my comment helps, please give it a thumbs up!
0 Karma
Reply

carbin
New Member
‎03-21-2019 05:19 AM

Thanks for the suggestion. I executed that PS script as admin, rebooted and still won't install. I may have to see if I can revert to an earlier snapshot of the VM (if one exists), but I'm unsure what I would do differently. Will the 7.2.5 installed upgrade or install over a 7.2.4 install?

0 Karma
Reply

carbin
New Member
‎03-20-2019 10:27 AM

In desperation I went to my other test server and did the following:

1: Verified that Add/Remove Programs did NOT contain Universal Forwarder
2: Moved the original install folder to a new Temp folder
3: Opened Regedit and searched for "splunk"
4: Deleted all that matched
5: Rebooted server
6: Attempted Installation. It failed and rolled back.

Did I miss Registry Keys?

0 Karma
Reply

carbin
New Member
‎03-20-2019 08:15 AM

cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T09:20:54-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.
11:12:37 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" _internal first-time-run --answer-yes --no-prompt >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=7.2.5
BUILD=088f49762779
PRODUCT=splunk
PLATFORM=Windows-AMD64

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.

"N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem": already a renewed Splunk certificate: skipping renewal
"N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem": already a renewed Splunk certificate: skipping renewal
[App Key Value Store migration] Binary for service(34) is missing.
[App Key Value Store migration] Binary for service(34) is missing.

-- Migration information is being logged to 'N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.11-12-38' --
11:12:41 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" _internal pre-flight-checks --answer-yes --no-prompt >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from 'N:\Program Files\SplunkUniversalForwarder\splunkforwarder-7.2.5-088f49762779-windows-64-manifest'
All installed files intact.
Done
11:12:45 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd install --startup=auto >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Installing service SplunkForwarder
Service installed
11:12:46 AM
cmd.exe /c "icacls "N:\Program Files\SplunkUniversalForwarder\etc" /T /C /grant *S-1-5-32-544:f >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
processed file: N:\Program Files\SplunkUniversalForwarder\etc
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth
processed file: N:\Program Files\SplunkUniversalForwarder\etc\copyright.txt
processed file: N:\Program Files\SplunkUniversalForwarder\etc\datetime.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\deployment-apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\disabled-apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-btool-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-btool.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-cmdline-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-cmdline.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-debug.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log-utility.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\log.cfg
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall
processed file: N:\Program Files\SplunkUniversalForwarder\etc\passwd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\prettyprint.xsl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk-launch.conf.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\splunk.version
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system
processed file: N:\Program Files\SplunkUniversalForwarder\etc\users
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\bin
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\bin\collector.path
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\introspection_generator_addon\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\learned\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\restmap.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\transforms.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\search\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\default-mode.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\limits.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\web.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\metadata\local.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_httpinput\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth.rnd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\appsCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\appsLicenseCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\ca.srl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cacert.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\cloudCA.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\crl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\server.pem
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\splunk.secret
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\crl\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release\ca.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\auth\prev_release\cacert.pem.default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\deployment-apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\disabled-apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses\forwarder
processed file: N:\Program Files\SplunkUniversalForwarder\etc\licenses\forwarder\splunkforwarder.lic
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\parsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\exec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\fschangemanager
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\RemoteQueue
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\structuredparsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\tailfile
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\TCP
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\UDP
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\wineventlog
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\winparsing
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\exec\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\fschangemanager\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\RemoteQueue\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\structuredparsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\tailfile\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\TCP\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\UDP\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\wineventlog\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\input\winparsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\modules\parsing\config.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\myinstall\splunkd.xml.cfg-default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\apps
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\users
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\apps\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\shcluster\users\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\admon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\MonitorNoHandle.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\perfmon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\powershell.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\powershell2.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\winEventLog.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinHostMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinNetMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinPrintMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\bin\WinRegMon.cmd
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\alert_actions.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\app.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\audit.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\authentication.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\authorize.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\conf.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\default-mode.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\health.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\limits.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\literals.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\livetail.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\messages.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\procmon-filters.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\props.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\restmap.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\source-classifier.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\sourcetypes.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\telemetry.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\visualizations.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\web.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\workload_pools.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\default\workload_rules.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\authentication.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\migration.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\README
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\local\user-seed.conf
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata\default.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\metadata\local.meta
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\alert_actions.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\alert_actions.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\audit.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\audit.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authentication.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authentication.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authorize.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\authorize.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\checklist.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\collections.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\collections.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\conf_checker.rules
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default-mode.conf.examples
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default-mode.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default.meta.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\default.meta.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\deploymentclient.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\deploymentclient.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\health.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\health.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\inputs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\inputs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\instance.cfg.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\instance.cfg.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\limits.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\limits.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\literals.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\literals.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\livetail.conf.examples
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\livetail.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\messages.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\messages.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\migration.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\outputs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\outputs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\passwords.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\passwords.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\procmon-filters.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\procmon-filters.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\props.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\props.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\restmap.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\restmap.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\server.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\server.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\serverclass.seed.xml.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\source-classifier.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\source-classifier.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\sourcetypes.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\sourcetypes.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\splunk-launch.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-prefs.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-prefs.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-seed.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\user-seed.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\visualizations.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\web.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\web.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\wmi.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\wmi.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_pools.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_pools.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_rules.conf.example
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\README\workload_rules.conf.spec
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static\atom.xsl
processed file: N:\Program Files\SplunkUniversalForwarder\etc\system\static\splunkrc_cmds.xml
processed file: N:\Program Files\SplunkUniversalForwarder\etc\users\users.ini
Successfully processed 230 files; Failed processing 0 files
11:12:46 AM
cmd.exe /c "icacls "N:\Program Files\SplunkUniversalForwarder\var" /T /C /grant *S-1-5-32-544:f >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
processed file: N:\Program Files\SplunkUniversalForwarder\var
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib
processed file: N:\Program Files\SplunkUniversalForwarder\var\log
processed file: N:\Program Files\SplunkUniversalForwarder\var\run
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\authDb
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\hashDb
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinputs
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\fishbucket\rawdata
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\modinputs\WinEventLog
processed file: N:\Program Files\SplunkUniversalForwarder\var\lib\splunk\persistentstorage\audit
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\introspection
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\watchdog
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\first_install.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.09-01-52
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.09-20-44
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\migration.log.2019-03-20.11-12-38
processed file: N:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log
processed file: N:\Program Files\SplunkUniversalForwarder\var\run.rnd
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\cachemanager_upload.json
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\composite.xml
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\upload
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\i18n
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules\static
processed file: N:\Program Files\SplunkUniversalForwarder\var\run\splunk\appserver\modules\static\css
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool\dirmoncache
processed file: N:\Program Files\SplunkUniversalForwarder\var\spool\splunk
Successfully processed 35 files; Failed processing 0 files
11:12:46 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 15:12:48 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T11:12:48-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:12:48 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 15:12:48 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:12:48 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.

0 Karma
Reply

nickhills
Ultra Champion
‎03-20-2019 08:50 AM

If you go to add-remove programs is the Splunk UF now gone?
If not, remove it from add/remove etc
Then move the folder from program files, and reinstall again.
It seems maybe the old installation directory is causing some odd behavior.

If my comment helps, please give it a thumbs up!
0 Karma
Reply

nickhills
Ultra Champion
‎03-20-2019 07:27 AM

Have you looked in the %TEMP%/splunkInstall.log file - If the install is failing it should contain details on the reason why.

If my comment helps, please give it a thumbs up!
0 Karma
Reply

carbin
New Member
‎03-20-2019 10:27 AM

In desperation I went to my other test server and did the following:

1: Verified that Add/Remove Programs did NOT contain Universal Forwarder
2: Moved the original install folder to a new Temp folder
3: Opened Regedit and searched for "splunk"
4: Deleted all that matched
5: Rebooted server
6: Attempted Installation. It failed and rolled back.

Did I miss Registry Keys?

0 Karma
Reply

carbin
New Member
‎03-20-2019 09:20 AM

That is the IP of the new server. When I run that search, it only returns one Host, the Splunk Ent server. I'm thinking Registry Keys must be holding me up

0 Karma
Reply

carbin
New Member
‎03-20-2019 08:58 AM

So, I moved the entire SplunkUniversalForwarder folder from Program Files to a temp folder and tried again:

…
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /services/apps/local/SplunkUniversalForwarder/enable >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 200 OK
Date: Wed, 20 Mar 2019 15:56:25 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/services/apps/local
2019-03-20T11:56:25-04:00

<name>Splunk</name>

0
30
0

<s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:56:25 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 201 Created
Date: Wed, 20 Mar 2019 15:56:25 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 4439
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

/servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server
2019-03-20T11:56:25-04:00

<name>Splunk</name>

1
30
0

<title>165.112.254.26:9997</title>
<id>/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997</id>
<updated>1969-12-31T19:00:00-05:00</updated>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="alternate"/>
<author>
  <name>nobody</name>
</author>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="list"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/_reload" rel="_reload"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="edit"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997" rel="remove"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/allconnections" rel="allconnections"/>
<link href="/servicesNS/nobody/system/data/outputs/tcp/server/165.112.254.26%3A9997/disable" rel="disable"/>
<content type="text/xml">
  <s:dict>
    <s:key name="destHost">165.112.254.26</s:key>
    <s:key name="destIp">165.112.254.26</s:key>
    <s:key name="destPort">9997</s:key>
    <s:key name="eai:acl">
      <s:dict>
        <s:key name="app">system</s:key>
        <s:key name="can_change_perms">1</s:key>
        <s:key name="can_list">1</s:key>
        <s:key name="can_share_app">1</s:key>
        <s:key name="can_share_global">1</s:key>
        <s:key name="can_share_user">0</s:key>
        <s:key name="can_write">1</s:key>
        <s:key name="modifiable">1</s:key>
        <s:key name="owner">nobody</s:key>
        <s:key name="perms">
          <s:dict>
            <s:key name="read">
              <s:list>
                <s:item>*</s:item>
              </s:list>
            </s:key>
            <s:key name="write">
              <s:list>
                <s:item>*</s:item>
              </s:list>
            </s:key>
          </s:dict>
        </s:key>
        <s:key name="removable">1</s:key>
        <s:key name="sharing">system</s:key>
      </s:dict>
    </s:key>
    <s:key name="method">autobalance</s:key>
    <s:key name="sourcePort">8089</s:key>
    <s:key name="status">not_connected</s:key>
  </s:dict>
</content>

DS init failed: Deployment Server not available on a dedicated forwarder.
11:56:26 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.

0 Karma
Reply

nickhills
Ultra Champion
‎03-20-2019 09:16 AM

Is 165.112.254.26 the IP of the old server, or the new one?

I had been assuming that it was the old server?

If you run a search on your new Splunk server for:
index=_internal |stats count by host
Do you see your forwarders in the results?

If my comment helps, please give it a thumbs up!
0 Karma
Reply

carbin
New Member
‎03-20-2019 07:33 AM

This is a Snippet:

9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd rest --noauth POST /servicesNS/nobody/SplunkUniversalForwarder/data/outputs/tcp/server "name=165.112.254.26:9997" >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
HTTP/1.1 400 Bad Request
Date: Wed, 20 Mar 2019 13:20:54 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 170
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<msg type="ERROR">165.112.254.26:9997 forwarded-server already present</msg>

DS init failed: Deployment Server not available on a dedicated forwarder.
9:20:54 AM
cmd.exe /c ""N:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" cmd splunkd uninstall >> "C:\Users\NINDSS~1\AppData\Local\Temp\splunk.log" 2>&1"
Removing service SplunkForwarder
Service removed
Disabled.

0 Karma
Reply

nickhills
Ultra Champion
‎03-20-2019 07:55 AM

The last 3 lines:

Removing service SplunkForwarder
Service removed
Disabled.

Suggest that the uninstall completed. What is written to that log file if you reinstall again?

If my comment helps, please give it a thumbs up!
0 Karma
Reply
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...