How to collect Windows event logs that are not from .evtx or .evt files?

[stcrispan]

I'm trying to collect Windows events. Specifically, I'm trying to collect:

\\Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Operational
\\Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Diagnostic
\\Applications and Service Logs\Microsoft\Windows\WLAN-Driver\Analytic

So far, I've installed the Universal Forwarder, and made the changes to the inputs.conf file so that it will collect these events directly from the .evtx files:

[default]
host = SLATE

[WinEventLog://Application]
disabled = 0
index=tablets
sourcetype=tablet_App

[WinEventLog://Microsoft-Windows-WLAN-AutoConfig/Operational]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Op

[WinEventLog://Microsoft-Windows-WLAN-Autoconfig/Diagnostic]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Diag

[WinEventLog://Microsoft-Windows-WLAN-Driver/Analytic]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Analytic

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

So I started getting data, specifically data from Operational, but there was a problem. The problem is that the Event File from Diagnostic and Analytic are not .evtx files, they're not even .evt files. They're .etl files...trace log files. So I wasn't getting any data from them.

The second problem is that I can't get Windows Event manager to create .evtx files for Diagnostic or Analytic. This is a BIG problem because it means that I can't automatically scroll old data off, like I would with a bona fide .evtx file. It only will create an .etl file, and that means it's going to have a finite limit. Once it fills up, you're done logging.

So there is another problem. I'm capturing these log files in order to troubleshoot a problem with tablets...tablets who, for some unknown reason, abruptly lose network connectivity. That means I need a buffer where log events can be stored pending network re-connection. So I can't simply feed the .etl file straight to Splunk (or can I?).

So given this set of problems, can someone recommend a solution where:

...the events from \Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Diagnostic and \Applications and Service Logs\Microsoft\Windows\WLAN-Driver\Analytic to get to Splunk.

...the solution is robust, fault tolerant, and not too complicated because I can't afford to weigh down the tablets.

Yes, I realize that this is not necessarily a Splunk problem, it's more of a Windows problem...but I figure, I can't be the ONLY guy who's ever run across a situation where a flat text file is being created, but needs to somehow be redirected to the UF...without actually creating a real file that can be filled up. I figure it's more of a redirect, or funneling...PIPING...problem.

Any takers? Any suggestions?

[stcrispan]

Lowell...

What it means is that you can't feed it directly in to Splunk. Period.

UNLESS...and this is a big unless, you really REALLY have to want this...unless, you are willing to set up a method on your own, whereby:

1) set up the ETL file to accept events until it fills up (a size that you can set)
- This means you have to configure the properties of the ETL file so that it has a finite size and does not roll old events, and then Enable it
2) create your own custom process which copies the ETL file somewhere else
- There is no set and confirmed method to do this, so you'll have to figure out a method on your own, perhaps using a set time period within Windows Task Manager, or maybe a custom job which watches the file size and copies it as new events occur
- You'll need a method to make sure your custom process stays running...if it's not running, you get no new events!
- Your custom process must perform cleanup on the ETL file...when it fills up, it must be emptied or deleted and re-created
3) The file which is created by your custom process can then be consumed as a log file by Universal Forwarder.

This above "solution" was the suggested solution when the problem of reading ETL files was presented to the Splunk Dev team. They first suggested we review the Micro$oft KnowledgeBase article, here:

https://support.microsoft.com/en-us/kb/2488055

...which basically tells us exactly what you mentioned above: if you set up an .ETL event log file, you can't read it and also have it roll events simultaneously. Period. By design. Micro$oft's design.

So we found something which Splunk can't do...and it's not Splunk's fault, it's a design limitation of the OS. But if Splunk was really gung-ho on their claim of eating all Windows Event files, they'd not just suggest we attempt a custom solution, they'd create an additional module for the UF which does this. And who knows, maybe it's on their plan, somewhere.

[Lowell]

Note that when you attempt to point WinEventLog input to an .etl file, you end up with the following error: WinEventLogChannel::init: Init failed, unable to subscribe to the Windows Event Log channel 'microsoft-windows-dnsserver/analytical': errorCode=15009. According to MSDN, 15009 means ERROR_EVT_SUBSCRIPTION_TO_DIRECT_CHANNEL. "You cannot subscribe to an Analytic or Debug channel; the events for an Analytic or Debug channel go directly to a log file and cannot be subscribed to."
Not sure what that means exactly, but sounds like a bad day.

I too confirmed that converting from the .etl log format to .evtx doesn't make the problem go away.

[stcrispan]

Since I wrote this, I have embarked upon a long, arduous journey into the bowels of Micro$oft's Event Logging. And, I just want to mention this up front...approximately five weeks ago (end of August) I opened up a case with Micro$oft to get some assistance. Then I began trying to figure it out myself.

First, the ETL file. ETL files are binary. They can usually be decoded with a M$ WDK kit tool called TRACEVIEW. TRACEVIEW can be used to open and view, or even convert the contents of an ETL file, using a key file called a TMF file. TMF files are contained in PDB files, and PDB files are contained within M$ Symbol files. There are many Symbol files, in fact, for each build of Windows, there are a minimum of two: Checked and Retail. That means that if you don't know exactly what your build of Windows is, you could end up downloading multiple Symbol files, each one taking about 10 minutes to download and another 20 minutes to install.

The really fun part of this is even if you do know your version of Windows (mine was 6.3.9600.17809, found with the command "SLMGR /DLV"), you still might not be able to find the exact Symbol file match to your build. For me, there were roughly a half dozen or so Windows 8 builds, every version of Windows 8...starting at 8.0 Developer Preview all the way up through Windows 8.1--and Windows 8.1 had several versions in of itself. Keep in mind that each one of these versions has at least two Symbol files to download and install. Fortunately for all, the install files are all named with the build of Windows to which they are relevant.

By the way, just having the Symbol files is not enough. If you plan to get a copy of TRACEVIEW, you will need to download the entire debugging set for Windows 8, which means you must have Visual Studio 12, the WDK, and the SDK. You will probably need the Debug Universal Drivers, too. Even this would not be too onerous, if it weren't for the fact that Micro$oft has sabotaged the link. (Just TRY to download the Debugging tools for Windows 8.1.)

So, after having downloaded the only components I could (SDK and WDK), and 8.1 build 9600 Symbol files, I began to sift through them (using TraceView and TracePDB) attempting to locate the GUID that the ETL file was encoded with. It wasn't there. I widened my search to previous builds, and even to the ARM builds. Not there. This entire effort took me the better part of a month.

This entire time I was playing tag with the M$ tech rep. I finally wrote to his team lead and his manager, and got a call almost immediately afterwards. He told me a quick, expedient way to turn my ETL file into an EVTX file. By going in to the properties of the specific event log, and changing the name of the file which the events are written to from ".etl" to ".evtx", it will save as a Windows Event Log file. It turns out that this entire time, when I had been requesting that he find and send to me the correct TMF file to unlock the ETL file, all I had to do was click my ruby slippers and say, "There's no file like .evtx...there's no file like .evtx..." Now here's the kicker: it turns out that this was by design ... according to him, there is no TMF file to decode these event logs.

The real bear of this situation is that I did that at the very beginning...and I got a text file with a different name but not a different file format.

So, to recap: The M$ technician told me how to create an .evtx file of the \\Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Diagnostic events. Examining the c:\windows\system32\winevt\logs directory for the Microsoft-Windows-WLAN-AutoConfig-Diagnostic.evtx file shows that it has the correct file extension and Windows is declaring it to be an authentic Windows Event Log file (whereas before it claimed it was just a text file).

Now that I have an authentic .evtx file, I should be able to use my Inputs.Conf just like it was, right? Well, I tried it, and it's not working. Just like before, I'm getting Application events, and I'm getting \\Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Operations events, but no \\Applications and Service Logs\Microsoft\Windows\WLAN-AutoConfig\Diagnostic events. I walked around my building, passing multiple AP's, watching the size of the Diagnostics.evtx file grow, and saw no events arrive at Splunk box. I went so far as to load Wireshark on the device, and watch the traffic going to the Splunk box. No Diagnostic events are being forwarded.

Here is my Inputs.Conf:

[default]
host = SLATE911KR

[WinEventLog://Application]
disabled = 0
index=tablets
sourcetype=tablet_App

[WinEventLog://Microsoft-Windows-WLAN-AutoConfig/Operational]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Op

[WinEventLog://Microsoft-Windows-WLAN-AutoConfig/Diagnostic]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Diag

[WinEventLog://Microsoft-Windows-WLAN-Driver/Analytic]
disabled=0
index=tablets
sourcetype=tablet_WLAN_Analytic


[WinEventLog://Security]
disabled = 1
index=tablets
sourcetype=tablet_Sec

[WinEventLog://System]
disabled = 1
index=tablets
sourcetype=tablet_Sys

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

I have checked the paths, and the file names, and made sure the case was correct...and even checked to see if case mattered, by changing the filename of the working .evtx file. Doesn't appear to have made any difference.

So for those Splunk experts who say, "If it's an .evtx file it can be forwarded by the Universal Forwarder and digested by Splunk.", my question is, what's next?

[woodcock]

Have you bounced the splunk service on the forwarding device? This is necessary to initiate changes in inputs.conf.

[stcrispan]

As I said, I think this is a dead-end...we know how to do it but given that it would be running on an HP tablet (limited CPU, limited RAM, limited storage...limited, period), we won't be pursuing it any further. It's just not time effective to use Splunk this way...not for ETL files, anyway.

And yes, I have bounced it. Lots.

[woodcock]

I had to ask. You are even more of a hero in my book, in that you bothered to post the full story for others even though it ended up not being of any benefit to yourself. BRAVO!

[woodcock]

I am SUPER impressed both by your tenacity to the task and your willingness to share your adventure is such creative prose! Surely the hard part is done and the Splunk part should be doable. I will do what I can to help, just PM me.

[stcrispan]

Thank you, sir. I think I have beat this horse to death...at this point, I'm just wearing my arm out.

[woodcock]

Something is going to have to decode it before it comes into splunk. Splunk has a way to facilitate this decoding automatically as part of the forwarding process but you must have access to the decoder tool as a command-line executable, for example:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff543596(v=vs.85).aspx

The process for this is well documented here:
http://blogs.splunk.com/2011/07/19/the-naughty-bits-how-to-splunk-binary-logfiles/