ERROR TcpInputProc - Message rejected. Received unexpected message of size=369296128 bytes for syslogs

[kundanshekhx]

Hi, 

I am trying to inboard a new Syslog coming from a Syslog ng server but data is not indexing.

Getting the below error in the internal logs in SH.

ERROR TcpInputProc - Message rejected. Received unexpected message of size=369296128 bytes from src=xx.xx.xx.xx:xxxxx in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

Below is the path I have set for the incoming logs.

Syslog-ng server > Universal Forwarder(TCP port) > Indexer

Below are the configurations set at the forwarder end:

inputs.conf
[tcp://xxxxx]
sourcetype=syslog
index = Index_name
disabled=false

outputs.conf
[tcpout]
defaultGroup = ABC
maxQueueSize = 7MB
useACK = true

[tcpout:ABC]
server = index_server1:42000, index_server2:42000, index_server3:42000

# SSL SETTINGS
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
sslPassword = xxxx
sslVerifyServerCert = true

After the issue, I have tried to resolve it by setting the value of bucketRebuildMemoryHint to auto and manually both in the indexes.conf but it didn't work.

indexes.conf

[default]

bucketRebuildMemoryHint = 569366123.

Can anyone please advise me on this?  Please let me know in case I am missing any information I missed to share which might help in reaching out to the solution.

Thanks in Advance 🙂 

[isoutamo]

Is this error message from your indexer or UF?

btw. when you are using useAck then maxQueueSize is automatic 7MB.

Do you know how big the message which are coming from syslog-ng is?

Does this maybe the reason? You have put some SSL configs on UF , but are indexers expecting SSL?

https://community.splunk.com/t5/Getting-Data-In/quot-ERROR-TcpInputProc-Message-rejected-quot-error-...

r. Ismo

[kundanshekhx]

Hi R.Ismo,

Thanks for the reply.

The error message is from the indexer.

As per the error message, the size of the incoming message is 369296128  bytes that turn around 352 MB.

SSL is working fine as we have logs from the other data sources coming to indexers through the same UF. 

This is the first time we are trying to inboard the Syslogs using TCP port. 

Thanks,

Kundan

[Vamsikrishna]

Hi  @kundanshekhx 

Did you fix this issue?

If yes, Please let me know how you fixed.

[PickleRick]

@VamsikrishnaThis is a rather old thread and the thread author's last activity on the forum is about 3 years ago so it's relatively unlikely you'll get answer from them.

To the main point - I'd guess that for one reason or another the forwarders fails to break the input stream into small enough chunks before sending it downstream.