Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protocol.

[hrawat]

See SPL-248479 in release notes.

If you are using persistent queue and see following errors in splunkd.log. 

ERROR TcpInputProc - Encountered Streaming S2S error

1. "Cannot register new_channel"

2. "Invalid payload_size"

3. "Too many bytes_used"

4. "Message rejected. Received unexpected message of size"

5. "not a valid combined field name/value type for data received"

Other S2S streaming errors as well.
 
You should upgrade your HF/IHF/IUF/IDX instance (if using persistent queue ) to following patches.
9.4.0/9.3.2/9.2.4/9.1.7 and above.

This patch also fixes all the known PQ related crashes and other PQ issues. 

[hrawat]

For customers hitting Cannot register new_channel error regardless of persistent queue at IF, 

applying 9.4.x/9.3.2/9.2.4/9.1.7 and above should fix the issue or reduce the chance of events entering  into splunkcloud DLQ.

[inderjot]

What should be plan for customers who recently upgraded to 9.3.3?

[hrawat]

9.3.3 is fine.
9.4.x/9.3.2/9.2.4/9.1.7 and above has the fix.

[AF_Ops]

@hrawat 
The email sent titled "Splunk Service Bulletin Notification" was very poorly written. It explicitly states to upgrade to one of the following versions, it doesn't say "or later".

We have recently upgraded all our forwarders to be running 9.4.1, which according to the service bulletin email isn't fixed, only 9.4.0 is (was there regression, or is the email wrong?).

[hrawat]

9.4.0/9.3.2/9.2.4/9.1.7 and above has the fix. Since you are already on 9.4.1, it also has the fix.

[inderjot]

Thanks for confirming

[edhealea]

Hey,
Email was release today from Splunk Cloud Platform Team stating  to fix this issue we should patch up to 9.4.0, 9.3.2, 9.2.4 or 9.1.7 as you have mentioned above.
Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
Second question,  If Splunk issued the recommendation to patch up to a higher level patch, why would they come back and recommend patch to a lower version with security vulnerabilities instead of patching up?

[hrawat]

>9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?

Yes.

>Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
I think the new advisory is just telling the fix is in 9.4.0, 9.3.2, 9.2.4 or 9.1.7 and above . However if you are already on 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions and above, you can ignore new email.