After upgrade, Splunk can no longer read evt files

[DaClyde]

We got stuck using 4.0.11 for a very long time, but during that time, it had no trouble importing exported Windows Event Logs in evt format. Now that we have upgraded through 4.2.5 to 4.3.2, Splunk will recognize Windows Event Log data from a forwarder, but when attempting to manually import an evt file, the preview show this:

Failed to decode 2155 bytes: source::C:\workbench\logs\jkreceive.evt|host::SPLUNK01|preprocess-winevt|

0\x00\x00\x00LfLe\x1\x00\x00\x00\x1\x00\x00\x000\x00\x00\x00D\xFE\xFF\x00\x8F\xEE\x15\x00\xF4R\x15\x00l\xFE\xFF\x00\x00\x00\x00\x00\x00\x00\x00\x000\x00\x00\x00\xD8\x1\x00\x00LfLe\xF4R\x15\x00߱aO߱aO\x00\x00\x00\x00\x4\x00\x1\x00\x00\x00\x00\x00\x00\x00\x00\x00d\x00\x00\x00\x00\x00\x00\x00d\x00\x00\x00\x00\x00\x00\x00\xD2\x1\x00\x00J\x00K\x00R\x00e\x00c\x00e\x00i\x00v\x00e\x00\x00\x00J\x00T\x00D\x00I\x00-\x00R\x00S\x00A\x00-\x000\x009\x00\x00\x00F\x00i\x00l\x00e\x00 \x00r\x00e\x00c\x00e\x00i\x00v\x00e\x00d\x00 \x00f\x00r

And all the "events" just get piled into a single day.

Events from the local machine's event log are imported fine. I'm seeing this both on a Win7 laptop I'm using for testing 4.3.1, as well as a Windows 2008 server running 4.3.1. Back when we were running 4.0.11, we had a data input set for a particular directory, and we would just drop exported evt files in there and Splunk had no trouble consuming them. We need that capability back. It doesn't seem to matter what OS the evt files are from as I've tried it with files exported from XP, 2k3, 7 and 2k8 and get the same results for both evt and evtx files.

What changed and how can we fix it?

[cofi_alan]

I'm having the same issue. Manual imports don't work but automated indexing is fine. Thought I'd make a note of it so that the Splunk support team might notice this thread.

[DaClyde · Accepted solution]

Ok I finally tracked down the culprit for the monitored folder. We had attempted to change the "sourcetype" to a manual setting, which Splunk couldn't interpret. Setting it back to "Automatic" cleared up the automated import issue. That setting changed sometime after our 4.2.5 upgrade but before the 4.3.1 upgrade.

The single file import still shows a mess in the preview, but we got the automated import working again.

[DaClyde]

Posting this as an answer to have more characters to work with.

The issue seems to be with the preview feature when attempting to manually import an individual file. In an attempt to see where this issue cropped up, I cleaned Splunk off my laptop, and re-installed 4.0.11. I created a data input watching an empty folder. When I dropped an evt file in there, it recognized it and parsed it properly into the index.

So I upgraded from 4.0.11 to 4.1.8, cleared the indexes and tried it again. Success.

So I upgraded from 4.1.8 to 4.2.5, and tried the same process, and again it read the file and properly showed the events.

So I upgraded from 4.2.5 to 4.3.2 and it STILL recognizes the files if I drop them into my watched folder.

However, neither 4.3.1, nor 4.3.2 have any idea what to do with an evt file if I attempt to import an individual file manually. In that scenario, I get the gibberish as I posted in the initial question.