Deployment Architecture

UF can't perform a handshake with DS that's behind an Apache reverse proxy


In our current setup we have a private network with several hosts that have UFs installed, as well as separate hosts for a search head, indexer and a Splunk Deployment server. Since All servers where Splunk UFs are installed are in the same private network, they have simply been set up to use the private IP of the deployment server in deploymentclient.conf.

We also have a separate host where a reverse proxy has been configured, using Apache. This host is the only server that has a public IP. The reverse proxy is used so that we can access the web UIs of Splunk search head and deployment server.

Until now this setup has been working well, but now I have to add another UF that is outside of the private network. To do this, I have added another configuration file to the reverse proxy that looks like this:

Listen 8089 https
<VirtualHost *:8089>

    ProxyPass "/"  "https://{ds_private_ip}:8089/"
    ProxyPassReverse "/"  "https://{ds_private_ip}:8089/"

    SSLEngine on
    SSLProxyEngine on
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLCertificateFile "/etc/letsencrypt/live/"
    SSLCertificateKeyFile "/etc/letsencrypt/live/"
    SSLCertificateChainFile "/etc/letsencrypt/live/"

The configuration itself seems to be working fine, as I can successfully connect to with a curl command from the server where the new UF is installed.

However after adding to deploymentclient.conf of the new UF, it can't perform a handshake. The most relevant part from splunkd.log of the UF seems this:

10-04-2017 05:33:37.379 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
10-04-2017 05:33:46.755 -0400 INFO  HttpPubSubConnection - SSL connection with id: connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}
10-04-2017 05:33:46.759 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /services/broker/channel/subscribe/connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}/tenantService/handshake/reply/{uf_hostname}/{uf_client_name} was not found on this server.</p>\n</body></html>\n

Where {rev_proxy_private_ip} is the private IP of the reverse proxy server; {rev_proxy_internal_dsn} is the internal DNS of the reverse proxy (since we're hosting everything on AWS, it's the one that looks like; {uf_hostname} is the hostname of the server with UF; {uf_client_name} is the client name configured in deploymentclient.conf.

So it seems that when the connection id is created, it's kind of mixing things from the UF server and the rev proxy server.
Since I'm not that experienced with web servers, I haven't been able to solve this.

Has anyone encountered a problem like this? Any suggestions for solving it?


same issue here

0 Karma


I think you might be running into issues trying to forward non-http traffic through an http proxy. Proxies are a bit out of my area so consider this answer a best-effort from a non-expert.

If the UF is using https and operates strictly through the REST API then I would expect it to work with a web proxy. If the UF uses some other application protocol, or there is something weird about how the UF uses https, then an http proxy would balk at the request because it could not parse what the UF is sending. I would expect a well-behaved proxy would return an HTTP error code, probably something in the 400-500 error code range. That might show up in the UF logs.

I do not know about Apache but I believe nginx can act as a TCP proxy. That approach might work, but it's not something I've used or researched in depth. I'd talk to your local proxy administrator for some expert information.

If you have the option, I would suggest skipping the proxy and manage the UF manually.

You'll also need to have a solution to cover log forwarding from UF to your internal Splunk servers (which happens using a proprietary protocol). I would recommend having a conversation with your networking folks about what options you have to pass the traffic Splunk needs to your private network. A NAT would be one option.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...