Installing a universal forwarder in low privilege mode, why am I getting error "Deployment Server not available on a dedicated forwarder"?

[lycollicott]

Our admin created me a regular domain user to test low P and assigned it these privileges:

• Permission to log on as a service. 
• Permission to log on as a batch job.
• Permission to replace a process-level token.
• Permission to act as part of the operating system.
• Permission to bypass traverse checking

I run this to test the automation:

msiexec /i splunkforwarder-6.3.2-aaff59bb082c-x64-release.msi AGREETOLICENSE=Yes INSTALLDIR=c:\SplunkUniversalForwarder RECEIVING_INDEXER=heavy.forwarder:9997 DEPLOYMENT_SERVER=deploy.server:8089 SET_ADMIN_USER=0 LOGON_USERNAME=DOMAIN\splunklpuser LOGON_PASSWORD=somethingclever /quiet /log  lar.txt

The lar.txt log shows a 1603 permissions error and the appdata\local\temp\splunk.log shows this as the failure point:

Deployment Server not available on a dedicated forwarder

The communication path to the deployment server is open and if I install with LocalSystem, then it is successful.

What is my DOMAIN\splunklpuser userid missing?

[lycollicott · Accepted solution]

It is definitely the super complex password with special charaters.

I resolved it in Powershell by escaping the entire password in single quotes.

LOGON_PASSWORD='somethingclever'

cmd.exe still does not like one of the special characters in the password, but we should be able to escape it with a carat.

It still isn't clear why that particular error message occurred when the problem was something else entirely.

[pellegrini]

DS init failed: Deployment Server not available on a dedicated forwarder.

This is not a real error on an Universal Forwarder. There is no Deployment Server on an Universal Forwarder. There is just a Deployment Client on the UF.  I think this event is show everytime the UF starts. This is how it looks in version 7.1.2 and 7.36, both Windows and Linux:

INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.

[lycollicott]

I'm on to something - not to be confused with "on something" either.

I threw LAUNCHSPLUNK=0 into the mix and tried to start the service manually after the installer completed. That resulted in a 1069 logon failure, so I went into the "Log On" tab on the service properties and pasted in the password I used on the command line. Magic time.

So, something about the totally unreadable, unimaginable and ridiculously unmemorizable password is the problem. I wrapped it in quotes on the command line and that made no difference, but I finally have some evidence to go on.

[dillardo_2]

@lycollicott , thanks for the post. We were having the same issue on a Server 2016 box and using LaunchSplunk=0 resolved it.

[jkat54]

Check this out too:

https://support.microsoft.com/en-us/kb/834484

Tells the possible causes of the 1603 and how to resolve.

"You may receive this error message if any one of the following conditions is true:
The folder that you are trying to install the Windows Installer package to is encrypted.
The drive that contains the folder that you are trying to install the Windows Installer package to is accessed as a substitute drive.
The SYSTEM account does not have Full Control permissions on the folder that you are trying to install the Windows Installer package to. You notice the error message because the Windows Installer service uses the SYSTEM account to install software."

[lycollicott]

We verified permissions with the admins multiple times and we don't see anything that explains the 1603.

[jkat54]

So you dont have bitlocker or any other form of encryption?

[lycollicott]

No. I think the 1603 was the parent message of a 1069 when the misinterpreted password was used.

[jkat54]

What version(s) of windows please? It looks like it cant bind to any ports. I cant find the GPO for port binding to save my life.

[lycollicott]

Windows 2012r2

[jkat54]

Hi,

Splunk used to discourage setting the deployment server during msiexec / installation on universal forwarders. It appears maybe they no longer allow it???

So here is what you need to do, same command minus the deployment server argument. Then you need to run

 c:\splunkuniversalforwarder\splunk.exe set deploy-poll deploymentserverHostOrIp:8089

The docs here say I'm crazy, and maybe I am:

http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/DeployaWindowsdfviathecommandline

Still give it a shot and let me know please.

[lycollicott]

I deployed many UFs remotely with the command line specifying the deployment server and they all worked fine as long as I let LocalSystem run the services. I only have this issue when specifying low P mode.

[masonmorales]

TBH I would recommend opening a support case: http://login.splunk.com/page/sso_redirect?type=portal

[lycollicott]

Oh, i do have a case open too.

[jkat54]

All right then I'm crazy bat stuff. Did you try the command without specifying the deployment server?

If it works, would it then make sense that you could automate a second command that sets the deployment server? I guess it's a workaround, but it's barely a new line of code.

[s2_splunk]

Can you share a couple more log messages from splunkd.log BEFORE the Deployment Server message is issued, please? This message should not appear in isolation, there should be others in the DC:DeploymentClient category.

Also, does your domain user have full access to the Splunk installation directory?

[lycollicott]

processed file: C:\SplunkUniversalForwarder\var\spool\dirmoncache
processed file: C:\SplunkUniversalForwarder\var\spool\splunk
Successfully processed 29 files; Failed processing 0 files
HTTP/1.1 200 OK
Date: Fri, 05 Feb 2016 20:09:06 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<?xml version="1.0" encoding="UTF-8"?>
<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-->
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <title>localapps</title>
  <id>/services/apps/local</id>
  <updated>2016-02-05T20:09:06+00:00</updated>
  <generator build="aaff59bb082c" version="6.3.2"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/services/apps/local/_new" rel="create"/>
  <link href="/services/apps/local/_reload" rel="_reload"/>
  <link href="/services/apps/local/_acl" rel="_acl"/>
  <opensearch:totalResults>0</opensearch:totalResults>
  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>
  <opensearch:startIndex>0</opensearch:startIndex>
  <s:messages>
    <s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>
  </s:messages>
</feed>
DS init failed: Deployment Server not available on a dedicated forwarder.

Yes, it has full access to that directory.

[s2_splunk]

Can you share a couple more log messages from splunkd.log BEFORE the Deployment Server message is issued, please? This message should not appear in isolation, there should be others in the DC:DeploymentClient category.

[masonmorales]

Can it make an outgoing connection to deploy.server:8089 via TCP?

[lycollicott]

Yes it does. That was the first thing I checked.