Is it an invalid source sending data to splunktcp port or valid source sending unsupported payload?

[Marco_Develops]

I am setting up TCP with TLS. Currently I have a Syslog Server sending data to my Splunk Instance but my Message is being rejected:

02-09-2022 11:15:13.039 -0800 ERROR TcpInputProc [3972 FwdDataReceiverThread] - Message rejected. Received unexpected message of size=1009989694 bytes from src=myserver.com:1571 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

Below is my Inputs.conf

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[splunktcp-ssl:514]
sourcetype = syslog

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = tls1.2
cipherSuite = ECDHE-RSA-AES256-GCM-SHA384
sslPassword = PASSWORD

Any help is appreciated

Thank you,

Marco

[PickleRick]

As always - check for connectivity from remote with openssl s_client

Do a network dump and see if the tls negotiation is properly proceeding and the payload is indeed delivered over an encrypted channel.

Splunk is very sensitive to small mistakes in tls configuration and sometimes you get completely different thing than what you wished for.

[Marco_Develops]

Hi  @PickleRick ,

I know that TLS is working to some extent because I had to fix and select the correct Ciphers on each end. Also, when I switch my Syslog Server to Just  TCP and Leave Splunk as TLS. I get the following message: 

02-10-2022 10:08:59.956 -0800 ERROR TcpInputProc [5848 FwdDataReceiverThread] - Error encountered for connection from src=myserver.com:1571. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Leading me to believe that I am at least making the handshake when I specify TLS on both ends.  Just to clarify, this data is coming from a Syslog Server that I am trying to send to My Splunk Enterprise(No Forwarder)

Thank You,
Marco

@PickleRick

[PickleRick]

Sometimes this error on the server's side, completely unintuitively means that the client was not able to properly load key/cert and tries to connect without encryption. Check your client's logs and see if you have any errors regarding your cert/key.

[Marco_Develops · Accepted solution]

After thoroughly reading the Inputs.Conf. I realized that  splunktcp / splunktcp:ssl  is for data being forwarded. In my case since data was not being forwarded the appropriate field is tcp-ssl:514.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Inputsconf#Data_distribution:

Below is my new working inputs.conf :

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[tcp-ssl:514]
sourcetype = syslog

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = tls1.2
cipherSuite = ECDHE-RSA-AES256-GCM-SHA384
sslPassword = PASSWORD

I appreciate all your input.

-Marco

[PickleRick]

Indeed. I so rarely use plain tcp inputs (regardless of whether ssl-enabled or not) that I completely missed that 🙂