ForwardedEvents ingestion broken after update to 9.1

[PickleRick]

This is an informational post rather than a question.

If you use WEF to gather logs from your infrastructure to a single point from which you pick them up with

[WinEventLog://ForwardedEvents]

You might notice that this input can stop working after you upgrade to 9.1.0 (or above).

The forwarder will log to splunkd.log errors about wrong event format

Invalid WEC content-format:'Events', for splunk-format = rendered_eventSee the description for the 'wec_event_format' setting at $SPLUNK_HOME/etc/system/README/inputs.conf.spec for more details

 If you go to the inputs.conf spec file (either in the readme directory or on the Splunk website) you'll find the wec_event_format parameter (which was not present in versions up to 9.0.6) which must correspond with the setting in the WEF subscription settings. If the wec_event_format is "wrong" (the most typical situation will be when the WEF subscription is created as Events and the UF uses the default rendered_event value) , you need to set

wec_event_format = raw_event

in your input definition.

[abpe]

It's actually worse.  Splunk doesn't allow you to set the wec_event_format to RenderedText if the channel name doesn't start with ForwardedEvents.

10-20-2023 12:49:20.893 +0200 ERROR ExecProcessor [6396 ExecProcessorSchedulerThread] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" WinEventCommonChannel - WinEventLogChannelBase::enumLocalWECSubscriptions: subscription:'Applocker' - Invalid WEC destination channel ACME-WEC-Workstations/Applocker for content format RenderedText. RenderedText format is supported only on ForwardedEvents or custom channels named ForwardedEvents-1, ForwardedEvents-2, etc.Consider creating custom channels as the destination log, or change the content format of the subscription to "Events". See the description for the 'wec_event_format' setting at $SPLUNK_HOME/etc/system/README/inputs.conf.spec for more details. 

Also you can't set wec_event_format as 'Events' for ForwardedEvents channel and forget about having mixed events in the same channel.

It's amazing how such a breaking change was introduced under the carpet.