Splunk TcpInputProc Error causing possible outage

[galwood]

We are currently getting the below errors on our splunk Indexers and are every few hours the splunk indexer cluster is placed in to quarantine by the Heavy Forwarders. We have two heavy forwarders inbound to 3 indexers in index cluster. Looking over the logs these are the only errors that we are receiving.

04-01-2019 23:41:19.706 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|263011": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:51090.
04-01-2019 23:41:19.963 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|273702": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-ip]:51086.
04-01-2019 23:41:44.349 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|235289": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:51022.
04-01-2019 23:41:48.991 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|1839654": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:59700.
04-01-2019 23:41:49.075 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|2761310": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:59702.
04-01-2019 23:42:18.862 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|1839654": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:59742.
04-01-2019 23:42:18.875 +0000 ERROR TcpInputProc - Encountered Streaming S2S error=Cannot register new_channel="source::[log location]|host::[hostname]|sourcetype::iis|1839654": desired_field_count=19 conflicts with existing_field_count=0 for data received from src=[HF-IP]:59744.

[hrawat]

See https://community.splunk.com/t5/Knowledge-Management/Persistent-queue-problems/td-p/703859

[mherbeck]

Try changing the sourcetype to ms:iis

I had the same issue and from looking through the deployment app I noticed that the sourcetype was forced as iis and the props was looking for ms:iis. I am assuming you are using the TA for Windows IIS as well.

[amartin6]

Same issue with us, were either of you able to resolve?

[timpacl]

We have the exact same issue but no solution for you. I wanted to add to the thread to perhaps bring this back into view.