Getting Data In

How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

Communicator

Hi,

I have a serious problem with logs.. some events (below 0.01%) have strange characters.
- such strange characters seems to not repeat (all are unique)
- there are no regional characters in events - OS has english language set
- it affects mainly Splunk processes, but not only
- for one selected host - there are "good" and "bad" process names. For example host ending with "PAL01" has hundreds of events with "splunk-powershell.exe" process name, and one with chinese artifacts.

My example search to find such hosts:

index="wineventlog" host=* (EventID=4688 OR EventID=861) NewProcessName=* SubjectUserName=* *splunk-pow* NOT *splunk-power*

What is going on? It is a serious problem, because we don't know how many artifacts are there and how to find them all.alt text

Splunk Employee
Splunk Employee

I suggest that you examine the actual event logs on one of the servers with the event viewer and see if it originates there. If the logs are OK I would re-install the UF on those servers.

0 Karma

Explorer

lukasz92 - did reinstalling work to eliminate the artifacts?

We're seeing the same thing. What versions were you running (our forwarders are on 6.3.2, indexers on 6.4.3)?

0 Karma

SplunkTrust
SplunkTrust

What is creating these event logs?

0 Karma

Communicator

I don't understand your question.

They are forwarded to indexers with input:

[WinEventLog://Security]
disabled = 0
current_only = 1
index = wineventlog
renderXml=true

0 Karma

SplunkTrust
SplunkTrust

Valid answers might include McAfee antivirus, custom GPOs, UAC, standard windows auditing when access is granted, custom software, etc.

0 Karma

Communicator

This is only WinEventLog. I catch the events with EventCode 4688.

0 Karma

SplunkTrust
SplunkTrust

Yeah doesnt make much sense unless there is some data corruption along the way... packet loss, pinched network cable, disk corruption, etc.

0 Karma

SplunkTrust
SplunkTrust

Hi lukasz92,

check the CHARSET = <string> in props.conf for this sourcetype on the universal forwarder. Maybe you need to adjust this in your case http://docs.splunk.com/Documentation/Splunk/6.4.0/Admin/Propsconf#GLOBAL_SETTINGS

cheers, MuS

Communicator

Changed also on forwarders. It didn't help.

0 Karma

Communicator

I changed it to CP1252, it didn't help - strange characters appear.

  • I observed that number of strange characters usually equals number of normal characters that should be in this place
0 Karma