topic Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk? in Getting Data In

How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92 — Wed, 11 May 2016 08:37:22 GMT

Hi,

I have a serious problem with logs.. some events (below 0.01%) have strange characters.
- such strange characters seems to not repeat (all are unique)
- there are no regional characters in events - OS has english language set
- it affects mainly Splunk processes, but not only
- for one selected host - there are "good" and "bad" process names. For example host ending with "PAL01" has hundreds of events with "splunk-powershell.exe" process name, and one with chinese artifacts.

My example search to find such hosts:

index="wineventlog" host=* (EventID=4688 OR EventID=861) NewProcessName=* SubjectUserName=* *splunk-pow* NOT *splunk-power*

What is going on? It is a serious problem, because we don't know how many artifacts are there and how to find them all.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

MuS — Wed, 11 May 2016 20:56:34 GMT

Hi lukasz92,

check the CHARSET = <string> in props.conf for this sourcetype on the universal forwarder. Maybe you need to adjust this in your case http://docs.splunk.com/Documentation/Splunk/6.4.0/Admin/Propsconf#GLOBAL_SETTINGS

cheers, MuS

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92 — Mon, 23 May 2016 08:30:26 GMT

I changed it to CP1252, it didn't help - strange characters appear.

I observed that number of strange characters usually equals number of normal characters that should be in this place
- Strange characters usually(but not always) end with something similar to 'r' - after reading URL it seems to be http://www.fileformat.info/info/unicode/char/ffff/index.htm - this character.
- Only values are broken, not tag names.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

jkat54 — Mon, 23 May 2016 14:00:37 GMT

What is creating these event logs?

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92 — Tue, 24 May 2016 08:22:43 GMT

Changed also on forwarders. It didn't help.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92 — Tue, 24 May 2016 08:24:41 GMT

I don't understand your question.

They are forwarded to indexers with input:

[WinEventLog://Security]
disabled = 0
current_only = 1
index = wineventlog
renderXml=true

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

jkat54 — Tue, 24 May 2016 12:33:54 GMT

Valid answers might include McAfee antivirus, custom GPOs, UAC, standard windows auditing when access is granted, custom software, etc.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92 — Wed, 25 May 2016 12:35:38 GMT

This is only WinEventLog. I catch the events with EventCode 4688.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

jkat54 — Wed, 25 May 2016 12:45:54 GMT

Yeah doesnt make much sense unless there is some data corruption along the way... packet loss, pinched network cable, disk corruption, etc.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

sjohnson_splunk — Wed, 25 May 2016 14:27:47 GMT

I suggest that you examine the actual event logs on one of the servers with the event viewer and see if it originates there. If the logs are OK I would re-install the UF on those servers.

Re: How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

buysse — Wed, 12 Oct 2016 15:35:10 GMT

lukasz92 - did reinstalling work to eliminate the artifacts?

We're seeing the same thing. What versions were you running (our forwarders are on 6.3.2, indexers on 6.4.3)?