topic Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco in Knowledge Management

Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protocol.

hrawat — Mon, 05 May 2025 14:26:29 GMT

If you are using persistent queue and see following errors in splunkd.log.

ERROR TcpInputProc - Encountered Streaming S2S error 1. "Cannot register new_channel" 2. "Invalid payload_size" 3. "Too many bytes_used" 4. "Message rejected. Received unexpected message of size" 5. "not a valid combined field name/value type for data received"

Other S2S streaming errors as well.

You should upgrade your HF/IHF/IUF/IDX instance (if using persistent queue ) to following patches.
9.4.0/9.3.2/9.2.4/9.1.7 and above.

This patch also fixes all the known PQ related crashes and other PQ issues.

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

edhealea — Tue, 22 Apr 2025 21:19:57 GMT

Hey,
Email was release today from Splunk Cloud Platform Team stating to fix this issue we should patch up to 9.4.0, 9.3.2, 9.2.4 or 9.1.7 as you have mentioned above.
Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
Second question, If Splunk issued the recommendation to patch up to a higher level patch, why would they come back and recommend patch to a lower version with security vulnerabilities instead of patching up?

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

inderjot — Wed, 23 Apr 2025 13:56:06 GMT

What should be plan for customers who recently upgraded to 9.3.3?

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

hrawat — Wed, 23 Apr 2025 15:18:06 GMT

9.3.3 is fine.
9.4.x/9.3.2/9.2.4/9.1.7 and above has the fix.

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

hrawat — Wed, 23 Apr 2025 15:23:41 GMT

>9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?

Yes.

>Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
I think the new advisory is just telling the fix is in 9.4.0, 9.3.2, 9.2.4 or 9.1.7 and above . However if you are already on 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions and above, you can ignore new email.

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

inderjot — Wed, 23 Apr 2025 15:27:07 GMT

Thanks for confirming

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

AF_Ops — Sun, 27 Apr 2025 22:23:25 GMT

@hrawat
The email sent titled "Splunk Service Bulletin Notification" was very poorly written. It explicitly states to upgrade to one of the following versions, it doesn't say "or later".

We have recently upgraded all our forwarders to be running 9.4.1, which according to the service bulletin email isn't fixed, only 9.4.0 is (was there regression, or is the email wrong?).

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

hrawat — Mon, 28 Apr 2025 10:49:18 GMT

9.4.0/9.3.2/9.2.4/9.1.7 and above has the fix. Since you are already on 9.4.1, it also has the fix.

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

hrawat — Mon, 05 May 2025 14:24:23 GMT

For customers hitting Cannot register new_channel error regardless of persistent queue at IF,

applying 9.4.x/9.3.2/9.2.4/9.1.7 and above should fix the issue or reduce the chance of events entering into splunkcloud DLQ.

topic Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco in Knowledge Management

Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protocol.

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

Re: Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protoco

For customers hitting Cannot register new_channel error regardless of persistent queue at IF, applying 9.4.x/9.3.2/9.2.4/9.1.7 and above should fix the issue or reduce the chance of events entering into splunkcloud DLQ.

For customers hitting Cannot register new_channel error regardless of persistent queue at IF,

applying 9.4.x/9.3.2/9.2.4/9.1.7 and above should fix the issue or reduce the chance of events entering into splunkcloud DLQ.