Knowledge Management

Forwarders blocking / Splunk Cloud Dead Letter Queue (DLQ), due to a Persistent Queue (PQ) problem with S2S protocol.

hrawat
Splunk Employee
Splunk Employee

See SPL-248479 in release notes.


If you are using persistent queue and see following errors in splunkd.log. 

 

ERROR TcpInputProc - Encountered Streaming S2S error

1. "Cannot register new_channel"

2. "Invalid payload_size"

3. "Too many bytes_used"

4. "Message rejected. Received unexpected message of size"

5. "not a valid combined field name/value type for data received"

 

Other S2S streaming errors as well.
 
You should upgrade your HF/IHF/IUF/IDX instance (if using persistent queue ) to following patches.
9.4.0/9.3.2/9.2.4/9.1.7 and above.

This patch also fixes all the known PQ related crashes and other PQ issues. 

Labels (1)
Tags (3)

hrawat
Splunk Employee
Splunk Employee

For customers hitting Cannot register new_channel error regardless of persistent queue at IF, 

applying 9.4.x/9.3.2/9.2.4/9.1.7 and above should fix the issue or reduce the chance of events entering  into splunkcloud DLQ.

0 Karma

inderjot
Explorer

What should be plan for customers who recently upgraded to 9.3.3?

0 Karma

hrawat
Splunk Employee
Splunk Employee

9.3.3 is fine.
9.4.x/9.3.2/9.2.4/9.1.7 and above has the fix.

AF_Ops
Observer

@hrawat 
The email sent titled "Splunk Service Bulletin Notification" was very poorly written. It explicitly states to upgrade to one of the following versions, it doesn't say "or later".

We have recently upgraded all our forwarders to be running 9.4.1, which according to the service bulletin email isn't fixed, only 9.4.0 is (was there regression, or is the email wrong?).

AF_Ops_0-1745792447509.png

 

0 Karma

hrawat
Splunk Employee
Splunk Employee

9.4.0/9.3.2/9.2.4/9.1.7 and above has the fix. Since you are already on 9.4.1, it also has the fix.

0 Karma

inderjot
Explorer

Thanks for confirming

0 Karma

edhealea
Path Finder

Hey,
Email was release today from Splunk Cloud Platform Team stating  to fix this issue we should patch up to 9.4.0, 9.3.2, 9.2.4 or 9.1.7 as you have mentioned above.
Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
Second question,  If Splunk issued the recommendation to patch up to a higher level patch, why would they come back and recommend patch to a lower version with security vulnerabilities instead of patching up?

hrawat
Splunk Employee
Splunk Employee

>9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?

Yes.

>Last month in the "Splunk Security Advisories" it said to patch up to 9.4.1, 9.3.3, 9.2.5, and 9.1.8 so if we are on the 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions, we are in the fix?
I think the new advisory is just telling the fix is in 9.4.0, 9.3.2, 9.2.4 or 9.1.7 and above . However if you are already on 9.4.1, 9.3.3, 9.2.5, and 9.1.8 versions and above, you can ignore new email.

Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...