In our current setup we have a private network with several hosts that have UFs installed, as well as separate hosts for a search head, indexer and a Splunk Deployment server. Since All servers where Splunk UFs are installed are in the same private network, they have simply been set up to use the private IP of the deployment server in deploymentclient.conf.
We also have a separate host where a reverse proxy has been configured, using Apache. This host is the only server that has a public IP. The reverse proxy is used so that we can access the web UIs of Splunk search head and deployment server.
Until now this setup has been working well, but now I have to add another UF that is outside of the private network. To do this, I have added another configuration file to the reverse proxy that looks like this:
Listen 8089 https
<VirtualHost *:8089>
ServerName ds.oursplunk.com
ProxyPass "/" "https://{ds_private_ip}:8089/"
ProxyPassReverse "/" "https://{ds_private_ip}:8089/"
SSLEngine on
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLCertificateFile "/etc/letsencrypt/live/ds.oursplunk.com/cert.pem"
SSLCertificateKeyFile "/etc/letsencrypt/live/ds.oursplunk.com/privkey.pem"
SSLCertificateChainFile "/etc/letsencrypt/live/ds.oursplunk.com/fullchain.pem"
</VirtualHost>
The configuration itself seems to be working fine, as I can successfully connect to https://ds.oursplunk.com:8089/ with a curl command from the server where the new UF is installed.
However after adding ds.oursplunk.com:8089 to deploymentclient.conf of the new UF, it can't perform a handshake. The most relevant part from splunkd.log of the UF seems this:
10-04-2017 05:33:37.379 -0400 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
10-04-2017 05:33:46.755 -0400 INFO HttpPubSubConnection - SSL connection with id: connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}
10-04-2017 05:33:46.759 -0400 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /services/broker/channel/subscribe/connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}/tenantService/handshake/reply/{uf_hostname}/{uf_client_name} was not found on this server.</p>\n</body></html>\n
Where {rev_proxy_private_ip} is the private IP of the reverse proxy server; {rev_proxy_internal_dsn} is the internal DNS of the reverse proxy (since we're hosting everything on AWS, it's the one that looks like ip-XX-XX-XXX-XX.aws-region.compute.internal); {uf_hostname} is the hostname of the server with UF; {uf_client_name} is the client name configured in deploymentclient.conf.
So it seems that when the connection id is created, it's kind of mixing things from the UF server and the rev proxy server.
Since I'm not that experienced with web servers, I haven't been able to solve this.
Has anyone encountered a problem like this? Any suggestions for solving it?
... View more