topic Re: SSL connection between Indexer and Forwarder in Getting Data In

SSL connection between Indexer and Forwarder

garima_chauhan — Mon, 28 Sep 2020 15:14:44 GMT

Hi,

I am not able to configure the ssl connections between the forwarder and indexer. The splunkd logs on both the indexer and forwarder are not the same as cited in the documentation.

Here is what I get on Indexer in splunkd.log:
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is not compressed
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is compressed
Date Time +1100 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

After this, I do not get any other message as mentioned in the documentation.

On Forwarder, I get the following in splunkd.log:
Date Time +1100 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
Date Time +1100 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
Date Time +1100 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
Date Time +1100 INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
Date Time +1100 INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=512000 in bytes.

Date Time +1100 WARN TcpOutputProc - Connected to idx=:9997. Not using ACK.

Date Time +1100 INFO TcpOutputProc - Connection to :9997 closed. Connection closed

For enabling SSL connection between a forwarder and an indexer, I performed the following configurations:

On Indexer(Windows)
I added the following stanzas in $SPLUNK_HOME\etc\system\local\inputs.conf

[splunktcp-ssl:9997]
compressed = true

[SSL]
rootCA = $SPLUNK_HOME\etc\auth\cacert.pem
serverCert = $SPLUNK_HOME\etc\auth\server.pem
password = password

On Forwarder(Windows)
I added the following stanzas in $SPLUNK_HOME\etc\system\local\outputs.conf

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = :9997
sslCertPath = $SPLUNK_HOME\etc\auth\server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME\etc\auth\cacert.pem
sslVerifyServerCert = false

I am not able to figure out where am I making a mistake.Please help.

Re: SSL connection between Indexer and Forwarder

jtacy — Mon, 11 Nov 2013 22:57:24 GMT

The log on the indexer suggests that you might have two input.conf stanzas using port 9997. Can you verify that you don't have a regular splunktcp:9997 stanza in there? If you want to support both non-SSL and SSL forwarders, you'll need to choose a different port for either of the inputs. Here's working configuration from a forwarder and indexer:

Indexer

[splunktcp-ssl:9997]

[SSL]
password = password
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

Forwarder

[tcpout]
defaultGroup = indexer01

[tcpout:indexer01]
server = indexer01:9997
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem

I'm not using compression in my example since the documentation for inputs.conf says that it only applies to non-SSL inputs. I'd remove that from the indexer and forwarder config and restart both instances to make sure that's not part of the problem, but my main concern is the possibility that there are two listeners using the same port.

With the above configuration, you should start seeing events for your forwarder host in the _internal index almost right away. Good luck!

Re: SSL connection between Indexer and Forwarder

garima_chauhan — Tue, 12 Nov 2013 06:36:30 GMT

Hi jtacy,
No I dont have a regular splunktcp-9997 stanza already present in the inputs.conf. Still, I created and used a separate port-9996 and changed the config files as given by you. I am still not getting the desired output. While the logs on indexer are same, the forwarder splunkd logs also give the following line:

Date Time +1100 WARN TcpOutputProc - Cooked connection to ip=IndexerIP:9996 timed out

The data from forwarder is successfully being indexed in the indexer, though. Please suggest what else can be the problem here.

Re: SSL connection between Indexer and Forwarder

jtacy — Tue, 12 Nov 2013 23:07:25 GMT

Unless the "timed out" message is appearing repeatedly in the logs, if the events are being indexed you might be OK. However, again, I was only able to match your indexer log output when I intentionally added duplicate inputs. My live indexers output only one set of TcpInputConfig events when starting:

10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is not compressed

Have you run "splunk btool inputs list" just to double-check that you don't have a duplicate?

Re: SSL connection between Indexer and Forwarder

garima_chauhan — Tue, 19 Nov 2013 04:48:54 GMT

Hi, I have no duplicate inputs.conf. In splunkd.log, it still doesn't show me the stated output, however,in metrics.log, it is giving that ssl=true.Would this mean, that ssl is enabled now?

Please help.

Re: SSL connection between Indexer and Forwarder

jtacy — Mon, 28 Sep 2020 15:20:45 GMT

Here's a good entry from metrics.log on an indexer. Note connectionType=cookedSSL and ssl=true:

11-21-2013 18:21:04.211 -0600 INFO Metrics - group=tcpin_connections, 10.10.10.10:57308:9997, connectionType=cookedSSL, sourcePort=57308, sourceHost=10.10.10.10, sourceIp=10.10.10.10, destPort=9997, _tcp_Bps=157.95, _tcp_KBps=0.15, _tcp_avg_thruput=0.15, kb=2.31, _tcp_Kprocessed=2.31, _tcp_eps=0.27, build=163460, version=5.0.3, os=Windows, arch=x64, hostname=forwarder, guid=ABCDEF12-1234-ABCD-1234-ABCDEF123456, fwdType=uf, ssl=true, lastIndexer=None, ack=false

Re: SSL connection between Indexer and Forwarder

jtacy — Fri, 22 Nov 2013 00:30:07 GMT

If you want to be really sure, you could use Wireshark to capture traffic between the forwarder and indexer on your SSL port. Capture for a couple of minutes to make sure you see several connections (assuming 30s connect interval). Follow one of the TCP streams and the only text that should be readable includes things like SplunkCommonCA that pertain to the default certs.

This, combined with the info from metrics.log on the indexer, should give you pretty good confidence that SSL is working properly. If you're that concerned about SSL, make sure you're using a custom CA and verifying certs.

Re: SSL connection between Indexer and Forwarder

jtacy — Fri, 22 Nov 2013 00:31:36 GMT

Couple of links about using custom certs:
http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates
http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcertificates

Re: SSL connection between Indexer and Forwarder

garima_chauhan — Fri, 22 Nov 2013 07:12:19 GMT

Thanks jtacy. I am able to see the metrics.log as you mentioned.