ForwardedEvents ingestion broken after update to 9.1

[PickleRick]

This is an informational post rather than a question.

If you use WEF to gather logs from your infrastructure to a single point from which you pick them up with

[WinEventLog://ForwardedEvents]

You might notice that this input can stop working after you upgrade to 9.1.0 (or above).

The forwarder will log to splunkd.log errors about wrong event format

Invalid WEC content-format:'Events', for splunk-format = rendered_eventSee the description for the 'wec_event_format' setting at $SPLUNK_HOME/etc/system/README/inputs.conf.spec for more details

 If you go to the inputs.conf spec file (either in the readme directory or on the Splunk website) you'll find the wec_event_format parameter (which was not present in versions up to 9.0.6) which must correspond with the setting in the WEF subscription settings. If the wec_event_format is "wrong" (the most typical situation will be when the WEF subscription is created as Events and the UF uses the default rendered_event value) , you need to set

wec_event_format = raw_event

in your input definition.

[jeremyhewitt]

I have wasted so many hours trying to troubleshoot why my ForwardedEvents were not being ingested into the index.

Thank you, this fixed the issue.

The formatting of the search is very different though, and not all fields are showing up in the results; not sure why.

Edit: So how can I get new ingested events to look the same? And have the same fields?

E.g. I'm only using Splunk to ingest forwarded applocker logs. I can't display fields for publisher or file path for newly ingested events. They only show up for old ones that were ingested before the issue.

Edit 2: Fixed it I think by adding this line back in:

renderXML = 1

[abpe]

It's actually worse.  Splunk doesn't allow you to set the wec_event_format to RenderedText if the channel name doesn't start with ForwardedEvents.

10-20-2023 12:49:20.893 +0200 ERROR ExecProcessor [6396 ExecProcessorSchedulerThread] - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" WinEventCommonChannel - WinEventLogChannelBase::enumLocalWECSubscriptions: subscription:'Applocker' - Invalid WEC destination channel ACME-WEC-Workstations/Applocker for content format RenderedText. RenderedText format is supported only on ForwardedEvents or custom channels named ForwardedEvents-1, ForwardedEvents-2, etc.Consider creating custom channels as the destination log, or change the content format of the subscription to "Events". See the description for the 'wec_event_format' setting at $SPLUNK_HOME/etc/system/README/inputs.conf.spec for more details. 

Also you can't set wec_event_format as 'Events' for ForwardedEvents channel and forget about having mixed events in the same channel.

It's amazing how such a breaking change was introduced under the carpet.

[goran_epl]

It also does not work for me. We had 8.2.6 UF version and upgraded to 9.1.7. We also tried with versions 9.0.9, 9.2.4 and 9.3.2.

Regardless of the wec_event_format = raw_event , we still have errors in the log 

Invalid WEC content-format:'Events', for splunk-format = rendered_eventSee the description for the 'wec_event_format' setting at $SPLUNK_HOME/etc/system/README/inputs.conf.spec for more details.

And the data is not coming in.

[marnall]

Are you absolutely sure that your forwarded events are all raw_event and not rendered_event? I had this issue where my event collector was forwarding mixed logs. You must check the event collector and make sure all forwarded events are of the same format.

[goran_epl]

You were correct, this solved the issue