Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protocol.

[hrawat]

See SPL-248479 in release notes.

If you are using persistent queue and see following errors in splunkd.log. 

ERROR TcpInputProc - Encountered Streaming S2S error

1. "Cannot register new_channel"

2. "Invalid payload_size"

3. "Too many bytes_used"

4. "Message rejected. Received unexpected message of size"

5. "not a valid combined field name/value type for data received"

Other S2S streaming errors as well.
 
You should upgrade your HF/IHF/IUF/IDX instance (if using persistent queue ) to following patches.
9.4.x/9.3.2/9.2.4/9.1.7 and above.

This patch also fixes all the known PQ related crashes and other PQ issues. 

[inderjot]

What should be plan for customers who recently upgraded to 9.3.3?

[hrawat]

9.3.3 is fine.
9.4.x/9.3.2/9.2.4/9.1.7 and above has the fix.

[inderjot]

Thanks for confirming

[edhealea]

Hey,
Email was release today from Splunk Cloud Platform Team stating  to fix this issue we should patch up to 9.4.0, 9.3.2, 9.2.4 or 9.1.7 as you have mentioned above.
Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
Second question,  If Splunk issued the recommendation to patch up to a higher level patch, why would they come back and recommend patch to a lower version with security vulnerabilities instead of patching up?

[hrawat]

>9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?

Yes.

>Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
I think the new advisory is just telling the fix is in 9.4.0, 9.3.2, 9.2.4 or 9.1.7 and above . However if you are already on 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions and above, you can ignore new email.