Security

Is it an invalid source sending data to splunktcp port or valid source sending unsupported payload?

Marco_Develops
Path Finder

I am setting up TCP with TLS. Currently I have a Syslog Server sending data to my Splunk Instance but my Message is being rejected:

 

 

02-09-2022 11:15:13.039 -0800 ERROR TcpInputProc [3972 FwdDataReceiverThread] - Message rejected. Received unexpected message of size=1009989694 bytes from src=myserver.com:1571 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

 

 

 

Below is my Inputs.conf

 

 

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[splunktcp-ssl:514]
sourcetype = syslog

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = tls1.2
cipherSuite = ECDHE-RSA-AES256-GCM-SHA384
sslPassword = PASSWORD


 

 

 

Any help is appreciated

Thank you,

Marco

Labels (3)
0 Karma
1 Solution

Marco_Develops
Path Finder

After thoroughly reading the Inputs.Conf. I realized that  splunktcp / splunktcp:ssl  is for data being forwarded. In my case since data was not being forwarded the appropriate field is tcp-ssl:514.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Inputsconf#Data_distribution:



Below is my new working inputs.conf :

 

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[tcp-ssl:514]
sourcetype = syslog

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = tls1.2
cipherSuite = ECDHE-RSA-AES256-GCM-SHA384
sslPassword = PASSWORD

 

I appreciate all your input.

-Marco

 

View solution in original post

0 Karma

PickleRick
SplunkTrust
SplunkTrust

As always - check for connectivity from remote with openssl s_client

Do a network dump and see if the tls negotiation is properly proceeding and the payload is indeed delivered over an encrypted channel.

Splunk is very sensitive to small mistakes in tls configuration and sometimes you get completely different thing than what you wished for.

Marco_Develops
Path Finder

Hi  @PickleRick ,

I know that TLS is working to some extent because I had to fix and select the correct Ciphers on each end. Also, when I switch my Syslog Server to Just  TCP and Leave Splunk as TLS. I get the following message: 

 

02-10-2022 10:08:59.956 -0800 ERROR TcpInputProc [5848 FwdDataReceiverThread] - Error encountered for connection from src=myserver.com:1571. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

 


Leading me to believe that I am at least making the handshake when I specify TLS on both ends.  Just to clarify, this data is coming from a Syslog Server that I am trying to send to My Splunk Enterprise(No Forwarder)

Thank You,
Marco

@PickleRick

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Sometimes this error on the server's side, completely unintuitively means that the client was not able to properly load key/cert and tries to connect without encryption. Check your client's logs and see if you have any errors regarding your cert/key.

Marco_Develops
Path Finder

After thoroughly reading the Inputs.Conf. I realized that  splunktcp / splunktcp:ssl  is for data being forwarded. In my case since data was not being forwarded the appropriate field is tcp-ssl:514.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Inputsconf#Data_distribution:



Below is my new working inputs.conf :

 

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

[tcp-ssl:514]
sourcetype = syslog

[SSL]
serverCert = C:\Program Files\Splunk\etc\auth\server.pem
sslVersions = tls1.2
cipherSuite = ECDHE-RSA-AES256-GCM-SHA384
sslPassword = PASSWORD

 

I appreciate all your input.

-Marco

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Indeed. I so rarely use plain tcp inputs (regardless of whether ssl-enabled or not) that I completely missed that 🙂

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...