Thanks, I'll do some testing with this! ... View more

I do know I don't need it for Splunk, my question is rather if it can be done. When it comes to why, it's simply put that our network setup is complex and the only safe way to ensure non RFC-1918 collision is to use Public IPs. So the question comes down to; - Use public IPs on the entire setup? - Or to use VIP to minimize the usage of public addresses down to a single address instead of X amounts? ... View more

Thanks, just so I am sure I properly understand this: splunkforwarder -> indexers # this is forwarder-polling/initiated on port 9997. splunkforwarder -> deployment # this is also forwarder-polling/initiated, port 8089. indexer/deployment/head -> license server # this is indexer/deployment/head-polling on port 8089. So overall, splunkforwarders initiates every connection to their mothership. And splunk servers initiates connections to the license server. ... View more

If I understand this correctly, clustered indexers should not be configured with deployment server (DS)? And they need to be done with cluster master/manager (CM)? A total of three hosts/nodes: head, index1 and index2. My theory was that having head as the "master" doing everything aside from indexing was a logical move. I am not sure if there's some place in the web interface or elsewhere to see a list of all active roles a node has. Each node has its own installation under /opt/splunk/, but configured with different roles on the installation. I am not sure if I'm getting all the abbreviations correct. The setup is not yet in production - I want it fully operational before I declare it ready. ... View more

Forgot to mention that - FQDNs are used, but not included here. They should be clustered - if I've configured that correctly, I am not sure. I've used this on the indexers: splunk edit cluster-config -mode slave -master_uri (...) And this on head: splunk edit cluster-config -mode searchhead -master_uri [clustering] master_uri = https://head_fqdn:8089 mode = slave pass4SymmKey = <string> ~/etc/system/local/server.conf @ index1 and 2 Running 'splunk list cluster-peers' on head tells be that index1 has status "Up", but index2 is not listed. The /local/ path is currently used as, well, it worked. I would like to follow best practices though - 'TA' - what's that standing for? I have a basic understanding of some of the components, but am far from fluent. ... View more

Hi! First off, thanks for the reply! The "head" server contains everything except the indexing part.  I checked the link you provided, but can't see any issues with the networking so far. Index1 [splunk@index1 ~]# splunk btool --debug deploymentclient list /opt/splunk/etc/system/local/deploymentclient.conf [target-broker:deploymentServer] /opt/splunk/etc/system/local/deploymentclient.conf targetUri = head:9887 [splunk@index1 ~]# nc -vz head 9887 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to head_ip:9887. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. Index2 [splunk@index2 ~]$ splunk btool --debug deploymentclient list /opt/splunk/etc/system/local/deploymentclient.conf [target-broker:deploymentServer] /opt/splunk/etc/system/local/deploymentclient.conf targetUri = head:9887 [splunk@index2 ~]$ nc -vz head 9887 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to index2_ip:9887. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.   If the same is attempted with OpenSSL, this is however the result (or a small part of it anyway): Certificate chain 0 s:/CN=SplunkServerDefaultCert/O=SplunkUser i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com Verify return code: 19 (self signed certificate in certificate chain) I'm using certificates from another CA (defined in several .conf files). In server.conf, I've got serverCert (.pem) and sslRootCAPath (.pem), along with sslPassword (hashed password). The Splunk CA isn't mentioned in either of these. With some investigation, the certificate that's used here, is in the files located here: /opt/splunk/etc/auth/ Mine are located here: /opt/splunk/etc/apps/ssl/ Where the default SSL is used I see the default ones listed in several files, although I'm not sure how many are of relevance. /opt/splunk/etc/system/default/server.conf:caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem I believe this has priority under /local/, thus not relevant. README dirs seems irrelevant too. This one might be of importance, although I'm not sure when it's generated/updated, and if it's collecting info from server.conf or another file. /opt/splunk/var/run/splunk/merged/server.conf:caCertFile = $SPLUNK_HOME/etc/auth/cacert.pem   This got a bit long, mainly to (hopefully) avoid misunderstandings. I'm rather new to this and may be way off in my thinking here.. ... View more

The error experienced looks similar to this: Waiting for web server at https://127.0.0.1:8000 to be available...(....) WARNING: web interface does not seem to be available! splunkd.log contains entries like this: 08-03-2020 13:04:49.883 +0200 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr: 08-03-2020 13:04:49.884 +0200 INFO HttpPubSubConnection - Could not obtain connection, will retry after=68.922 seconds. 08-03-2020 13:04:59.474 +0200 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected All or most of these does however also appear on the servers where the web interface seems okay, and 'splunk restart' runs as expected. ... View more