After Upgrade from Splunk 7.2.3 to Splunk 8.0.1 we get error TcpInputProc - Encountered Streaming S2S error=Received reference to unknown channel_code=132

[QuintonS]

Hi,

I am dealing with an issue where after upgrading our Splunk environment from 7.2.3 to 8.0.1 we are having endless errrors as stated in the title on the indexers within the cluster.
Error - 01-23-2020 15:58:09.056 +0200 ERROR TcpInputProc - Encountered Streaming S2S error=Received reference to unknown channel_code=132 for data received from src=1

Data flow is - UF --> Heavy Forwarder --> Indexer

Anyone that can shed some light on this?

[QuintonS]

Thank you for the response, this solved my issue. Just another question is this only for the Heavy Forwarder to indexer or would it also be applicable from UF to Heavy Forwarder?

[yaasirvatham_sp · Accepted solution]

In the Heavy Forwarders, You have to go to $SPLUNK_HOME/etc/system/local/Outputs.conf and add the value "negotiateProtocolLevel = 0" under the stanza [tcpout] then restart Splunk service.

After you add that value in the configuration file, Splunk will start to use the old protocol to connect with indexers and the connection should be established again.

[tcpout]
negotiateProtocolLevel = 0

[hrawat_splunk]

The correct workaround should have been

[tcpout]
negotiateProtocolLevel = 5

 negotiateProtocolLevel = 0 is no longer valid (see enableOldS2SProtocol in 9.1.x outputs.conf)

with 9.1.x and is likely to cause issues.

[QuintonS]

Thank you for the response, this solved my issue. Just another question is this only for the Heavy Forwarder to indexer or would it also be applicable from UF to Heavy Forwarder?

[QuintonS]

I am asking since the Heavy Forwarders have also been upgraded to 8.0.1 but the UF's are still running 7.2.3 and are in the process of being upgraded.

[andreasz]

My Heavy Forwarders and Indexers are at version 8.0.2 and I still get the error. Why should we set the negotiateProtocolLevel to 0, if both servers (HF & Indexer) are already at the newest version?

[arcsight_guru]

Support confirmed that this is a bug (SPL-182112) for S2S communication between 8.x nodes. In my case I had issues between SH and INX. The recommendation was to set negotiateProtocolLevel=5 to downgrade the protocol version to 7.3. This can be done in the [tcpout] stanza on the sending node (SH), or in the [splunktcp] stanza on the receiving end (INX).

[jhomerlopez]

Hi, this was be solved on my environment by applying the below config on outputs.conf on your HeavyForwarder.

[tcpout]
negotiateProtocolLevel = 0

Once applied, you need to restart splunk service.