Receiving SSL data into a forwarder - ISAM9 request_syslogs to Splunk forwarder

[rewritex]

IBM Security Access Manager v9 build 9.0.1.0
* There is a bug which doesn't allow syslog to be sent of UDP, but TLS-TCP works. The bug is fixed in 9.0.2.0

On the ISAM9 side, within the proxy I have setup the logcfg parameter to send syslog out.

server-log-cfg = rsyslog server=10.10.10.10,port=10265,log_id=server01_msg_webseald-default.log,ssl_keyfile=default_qdsrv.kdb,ssl_stashfile=default_qdsrv.sth

On the Splunk Forwarder side: ( i send the logs to an intermediate forwarder which sends to the cluster)
In the Inputs.conf I have tried the variations - [tcp://:10265], [splunktcp-ssl://:10265], [tcp-ssl:10265] - switching out the : to ://: to //: since docs were not to clear.

When using splunktcp or tcp-ssl my splunkd.log (on the forwarder) reports these are reserved for Splunk2Splunk. Also, when I run netstat -apn | grep 10265 ... its not listening.

Question: I'm not sure if I generated a SSL cert correctly. I followed this link: https://answers.splunk.com/answers/130860/how-to-get-tcp-ssl-input-for-splunk-6-0-to-work.html but it can't find the genSignedServerCert.py file referenced in the script /opt/splunk/bin/genSignedServerCert.sh -d /opt/splunk/etc/certs -n splunk -c splunk -p so it fails.

Has anyone worked on this ISAM9 -> splunk forwarding?
Any accurate advice on howto receive SSL data into a forwarder?

Splunk 6.5.2
Splunk forwarder 6.4.3

Thank You,
Sean

[adhoke_splunk]

Look for [tcp-ssl] stanza in http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf. It should accept data from non-splunk inputs.

To check if certs are valid, verify with openssl :

openssl verify -CAfile [ca-bundle.crt] [certificate.crt]

Please post splunkd.log errors that you see.

[rewritex]

No port is opening on my server... SSL cert issues

Things of changed in version 6.5.2 so I updated the files using these links.
https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Inputsconf
https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Serverconf

I've reviewed the passwords and viewed the .pem cert files for a ----start---- line and i'm pretty positive things are correct.
For the passwords.... I stop the forwarder, input the clear_text password and restart ....
To verify LISTENING i do ... netstat -apn | grep 10265
I verified the path
I checked file permissions and they are setup correctly recursively.
I suppose I will try and recreate the certs making sure the passwords are correct

Server.conf:

[sslConfig]
sslRootCAPath = /<path_to_cert>/myCACertificate.pem
sslPassword = <password>

inputs.conf:

[tcp-ssl:10265]
_TCP_ROUTING = West01
index = isam9_0101
sourcetype = isam9

[SSL]
serverCert = /<path_to_cert>/myServerCertificate.pem
sslPassword = <password>

splunkd.log (from forwarder)

ERROR TcpInputConfig - SSL context not found. Will not open raw  (SSL) IPv4 port 10265
ERROR message = SSL context not found. Will not open raw (SSL) IPv4 port 10265

ERROR TcpInputConfig - SSL server certificate not found, or password is wrong - SSL ports will not be opened
ERROR message = SSL server certificate not found, or password is wrong - SSL ports will not be opened 

ERROR SSLCommon - Can't read key file /<path_to_cert>/myServerCertificate.pem errno=151441516 error:0906D06C:PEM routines:PEM_read_bio:no start line.

[adhoke_splunk]

Is the private key added to serverCert? if not, follow this:
https://answers.splunk.com/answers/55395/certificate-errors-for-forwarder.html

[rewritex]

I will need to double check if the key is in the cert.

The other issue that just started is as I modify the inputs.conf and server.conf ... Some of my other indexes stop receiving data and once I # comment out these new SSL entries, the indexes start working again. The inputs for these indexes use a standard [monitor://] input ...

My short-term goal is to accept TCP-SSL connections in the forwarder but keep the inter-splunk traffic non-SSL for now. I do plan to go change but not at thsi time