Getting Data In

How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92
Communicator

Hi,

I have a serious problem with logs.. some events (below 0.01%) have strange characters.
- such strange characters seems to not repeat (all are unique)
- there are no regional characters in events - OS has english language set
- it affects mainly Splunk processes, but not only
- for one selected host - there are "good" and "bad" process names. For example host ending with "PAL01" has hundreds of events with "splunk-powershell.exe" process name, and one with chinese artifacts.

My example search to find such hosts:

index="wineventlog" host=* (EventID=4688 OR EventID=861) NewProcessName=* SubjectUserName=* *splunk-pow* NOT *splunk-power*

What is going on? It is a serious problem, because we don't know how many artifacts are there and how to find them all.alt text

sjohnson_splunk
Splunk Employee
Splunk Employee

I suggest that you examine the actual event logs on one of the servers with the event viewer and see if it originates there. If the logs are OK I would re-install the UF on those servers.

0 Karma

buysse
Explorer

lukasz92 - did reinstalling work to eliminate the artifacts?

We're seeing the same thing. What versions were you running (our forwarders are on 6.3.2, indexers on 6.4.3)?

0 Karma

jkat54
SplunkTrust
SplunkTrust

What is creating these event logs?

0 Karma

lukasz92
Communicator

I don't understand your question.

They are forwarded to indexers with input:

[WinEventLog://Security]
disabled = 0
current_only = 1
index = wineventlog
renderXml=true

0 Karma

jkat54
SplunkTrust
SplunkTrust

Valid answers might include McAfee antivirus, custom GPOs, UAC, standard windows auditing when access is granted, custom software, etc.

0 Karma

lukasz92
Communicator

This is only WinEventLog. I catch the events with EventCode 4688.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yeah doesnt make much sense unless there is some data corruption along the way... packet loss, pinched network cable, disk corruption, etc.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi lukasz92,

check the CHARSET = <string> in props.conf for this sourcetype on the universal forwarder. Maybe you need to adjust this in your case http://docs.splunk.com/Documentation/Splunk/6.4.0/Admin/Propsconf#GLOBAL_SETTINGS

cheers, MuS

lukasz92
Communicator

Changed also on forwarders. It didn't help.

0 Karma

lukasz92
Communicator

I changed it to CP1252, it didn't help - strange characters appear.

  • I observed that number of strange characters usually equals number of normal characters that should be in this place
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...