UF can't perform a handshake with DS that's behind...

FoolMoon · ‎10-04-2017

In our current setup we have a private network with several hosts that have UFs installed, as well as separate hosts for a search head, indexer and a Splunk Deployment server. Since All servers where Splunk UFs are installed are in the same private network, they have simply been set up to use the private IP of the deployment server in deploymentclient.conf.

We also have a separate host where a reverse proxy has been configured, using Apache. This host is the only server that has a public IP. The reverse proxy is used so that we can access the web UIs of Splunk search head and deployment server.

Until now this setup has been working well, but now I have to add another UF that is outside of the private network. To do this, I have added another configuration file to the reverse proxy that looks like this:

Listen 8089 https
<VirtualHost *:8089>
    ServerName ds.oursplunk.com

    ProxyPass "/"  "https://{ds_private_ip}:8089/"
    ProxyPassReverse "/"  "https://{ds_private_ip}:8089/"

    SSLEngine on
    SSLProxyEngine on
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLCertificateFile "/etc/letsencrypt/live/ds.oursplunk.com/cert.pem"
    SSLCertificateKeyFile "/etc/letsencrypt/live/ds.oursplunk.com/privkey.pem"
    SSLCertificateChainFile "/etc/letsencrypt/live/ds.oursplunk.com/fullchain.pem"
</VirtualHost>

The configuration itself seems to be working fine, as I can successfully connect to https://ds.oursplunk.com:8089/ with a curl command from the server where the new UF is installed.

However after adding ds.oursplunk.com:8089 to deploymentclient.conf of the new UF, it can't perform a handshake. The most relevant part from splunkd.log of the UF seems this:

10-04-2017 05:33:37.379 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
10-04-2017 05:33:46.755 -0400 INFO  HttpPubSubConnection - SSL connection with id: connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}
10-04-2017 05:33:46.759 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /services/broker/channel/subscribe/connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}/tenantService/handshake/reply/{uf_hostname}/{uf_client_name} was not found on this server.</p>\n</body></html>\n

Where {rev_proxy_private_ip} is the private IP of the reverse proxy server; {rev_proxy_internal_dsn} is the internal DNS of the reverse proxy (since we're hosting everything on AWS, it's the one that looks like ip-XX-XX-XXX-XX.aws-region.compute.internal); {uf_hostname} is the hostname of the server with UF; {uf_client_name} is the client name configured in deploymentclient.conf.

So it seems that when the connection id is created, it's kind of mixing things from the UF server and the rev proxy server.
Since I'm not that experienced with web servers, I haven't been able to solve this.

Has anyone encountered a problem like this? Any suggestions for solving it?

secrecys · ‎01-10-2018

same issue here

jsrobard · ‎12-05-2017

I think you might be running into issues trying to forward non-http traffic through an http proxy. Proxies are a bit out of my area so consider this answer a best-effort from a non-expert.

If the UF is using https and operates strictly through the REST API then I would expect it to work with a web proxy. If the UF uses some other application protocol, or there is something weird about how the UF uses https, then an http proxy would balk at the request because it could not parse what the UF is sending. I would expect a well-behaved proxy would return an HTTP error code, probably something in the 400-500 error code range. That might show up in the UF logs.

I do not know about Apache but I believe nginx can act as a TCP proxy. That approach might work, but it's not something I've used or researched in depth. I'd talk to your local proxy administrator for some expert information.

If you have the option, I would suggest skipping the proxy and manage the UF manually.

You'll also need to have a solution to cover log forwarding from UF to your internal Splunk servers (which happens using a proprietary protocol). I would recommend having a conversation with your networking folks about what options you have to pass the traffic Splunk needs to your private network. A NAT would be one option.

UF can't perform a handshake with DS that's behind an Apache reverse proxy

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

Monitoring Amazon Elastic Kubernetes Service (EKS)