Getting Data In

Installing a universal forwarder in low privilege mode, why am I getting error "Deployment Server not available on a dedicated forwarder"?

Motivator

Our admin created me a regular domain user to test low P and assigned it these privileges:

• Permission to log on as a service. 
• Permission to log on as a batch job.
• Permission to replace a process-level token.
• Permission to act as part of the operating system.
• Permission to bypass traverse checking

I run this to test the automation:

msiexec /i splunkforwarder-6.3.2-aaff59bb082c-x64-release.msi AGREETOLICENSE=Yes INSTALLDIR=c:\SplunkUniversalForwarder RECEIVING_INDEXER=heavy.forwarder:9997 DEPLOYMENT_SERVER=deploy.server:8089 SET_ADMIN_USER=0 LOGON_USERNAME=DOMAIN\splunklpuser LOGON_PASSWORD=somethingclever /quiet /log  lar.txt

The lar.txt log shows a 1603 permissions error and the appdata\local\temp\splunk.log shows this as the failure point:

Deployment Server not available on a dedicated forwarder

The communication path to the deployment server is open and if I install with LocalSystem, then it is successful.

What is my DOMAIN\splunklpuser userid missing?

0 Karma
1 Solution

Motivator

It is definitely the super complex password with special charaters.

I resolved it in Powershell by escaping the entire password in single quotes.

LOGON_PASSWORD='somethingclever'

cmd.exe still does not like one of the special characters in the password, but we should be able to escape it with a carat.

It still isn't clear why that particular error message occurred when the problem was something else entirely.

View solution in original post

Motivator

It is definitely the super complex password with special charaters.

I resolved it in Powershell by escaping the entire password in single quotes.

LOGON_PASSWORD='somethingclever'

cmd.exe still does not like one of the special characters in the password, but we should be able to escape it with a carat.

It still isn't clear why that particular error message occurred when the problem was something else entirely.

View solution in original post

Motivator

I'm on to something - not to be confused with "on something" either.

I threw LAUNCHSPLUNK=0 into the mix and tried to start the service manually after the installer completed. That resulted in a 1069 logon failure, so I went into the "Log On" tab on the service properties and pasted in the password I used on the command line. Magic time.

So, something about the totally unreadable, unimaginable and ridiculously unmemorizable password is the problem. I wrapped it in quotes on the command line and that made no difference, but I finally have some evidence to go on.

0 Karma

Path Finder

@lycollicott , thanks for the post. We were having the same issue on a Server 2016 box and using LaunchSplunk=0 resolved it.

0 Karma

SplunkTrust
SplunkTrust

Check this out too:

https://support.microsoft.com/en-us/kb/834484

Tells the possible causes of the 1603 and how to resolve.

"You may receive this error message if any one of the following conditions is true:
The folder that you are trying to install the Windows Installer package to is encrypted.
The drive that contains the folder that you are trying to install the Windows Installer package to is accessed as a substitute drive.
The SYSTEM account does not have Full Control permissions on the folder that you are trying to install the Windows Installer package to. You notice the error message because the Windows Installer service uses the SYSTEM account to install software."

0 Karma

Motivator

We verified permissions with the admins multiple times and we don't see anything that explains the 1603.

0 Karma

SplunkTrust
SplunkTrust

So you dont have bitlocker or any other form of encryption?

0 Karma

Motivator

No. I think the 1603 was the parent message of a 1069 when the misinterpreted password was used.

0 Karma

SplunkTrust
SplunkTrust

What version(s) of windows please? It looks like it cant bind to any ports. I cant find the GPO for port binding to save my life.

0 Karma

Motivator

Windows 2012r2

0 Karma

SplunkTrust
SplunkTrust

Hi,

Splunk used to discourage setting the deployment server during msiexec / installation on universal forwarders. It appears maybe they no longer allow it???

So here is what you need to do, same command minus the deployment server argument. Then you need to run

 c:\splunkuniversalforwarder\splunk.exe set deploy-poll deploymentserverHostOrIp:8089

The docs here say I'm crazy, and maybe I am:

http://docs.splunk.com/Documentation/Splunk/6.3.3/Forwarding/DeployaWindowsdfviathecommandline

Still give it a shot and let me know please.

0 Karma

Motivator

I deployed many UFs remotely with the command line specifying the deployment server and they all worked fine as long as I let LocalSystem run the services. I only have this issue when specifying low P mode.

0 Karma

Influencer

TBH I would recommend opening a support case: http://login.splunk.com/page/sso_redirect?type=portal

0 Karma

Motivator

Oh, i do have a case open too.

0 Karma

SplunkTrust
SplunkTrust

All right then I'm crazy bat stuff. Did you try the command without specifying the deployment server?

If it works, would it then make sense that you could automate a second command that sets the deployment server? I guess it's a workaround, but it's barely a new line of code.

0 Karma

Splunk Employee
Splunk Employee

Can you share a couple more log messages from splunkd.log BEFORE the Deployment Server message is issued, please? This message should not appear in isolation, there should be others in the DC:DeploymentClient category.

Also, does your domain user have full access to the Splunk installation directory?

0 Karma

Motivator
processed file: C:\SplunkUniversalForwarder\var\spool\dirmoncache
processed file: C:\SplunkUniversalForwarder\var\spool\splunk
Successfully processed 29 files; Failed processing 0 files
HTTP/1.1 200 OK
Date: Fri, 05 Feb 2016 20:09:06 GMT
Expires: Thu, 26 Oct 1978 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Content-Type: text/xml; charset=UTF-8
X-Content-Type-Options: nosniff
Content-Length: 1930
Connection: Close
X-Frame-Options: SAMEORIGIN
Server: Splunkd

<?xml version="1.0" encoding="UTF-8"?>
<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-->
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <title>localapps</title>
  <id>/services/apps/local</id>
  <updated>2016-02-05T20:09:06+00:00</updated>
  <generator build="aaff59bb082c" version="6.3.2"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/services/apps/local/_new" rel="create"/>
  <link href="/services/apps/local/_reload" rel="_reload"/>
  <link href="/services/apps/local/_acl" rel="_acl"/>
  <opensearch:totalResults>0</opensearch:totalResults>
  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>
  <opensearch:startIndex>0</opensearch:startIndex>
  <s:messages>
    <s:msg type="INFO">Restart required by: default-mode, limits, server, web</s:msg>
  </s:messages>
</feed>
DS init failed: Deployment Server not available on a dedicated forwarder.

Yes, it has full access to that directory.

0 Karma

Splunk Employee
Splunk Employee

Can you share a couple more log messages from splunkd.log BEFORE the Deployment Server message is issued, please? This message should not appear in isolation, there should be others in the DC:DeploymentClient category.

0 Karma

Influencer

Can it make an outgoing connection to deploy.server:8089 via TCP?

0 Karma

Motivator

Yes it does. That was the first thing I checked.

0 Karma