Getting Data In

How to troubleshoot why we are seeing unexpected characters in Windows event logs in Splunk?

lukasz92
Communicator

Hi,

I have a serious problem with logs.. some events (below 0.01%) have strange characters.
- such strange characters seems to not repeat (all are unique)
- there are no regional characters in events - OS has english language set
- it affects mainly Splunk processes, but not only
- for one selected host - there are "good" and "bad" process names. For example host ending with "PAL01" has hundreds of events with "splunk-powershell.exe" process name, and one with chinese artifacts.

My example search to find such hosts:

index="wineventlog" host=* (EventID=4688 OR EventID=861) NewProcessName=* SubjectUserName=* *splunk-pow* NOT *splunk-power*

What is going on? It is a serious problem, because we don't know how many artifacts are there and how to find them all.alt text

sjohnson_splunk
Splunk Employee
Splunk Employee

I suggest that you examine the actual event logs on one of the servers with the event viewer and see if it originates there. If the logs are OK I would re-install the UF on those servers.

0 Karma

buysse
Explorer

lukasz92 - did reinstalling work to eliminate the artifacts?

We're seeing the same thing. What versions were you running (our forwarders are on 6.3.2, indexers on 6.4.3)?

0 Karma

jkat54
SplunkTrust
SplunkTrust

What is creating these event logs?

0 Karma

lukasz92
Communicator

I don't understand your question.

They are forwarded to indexers with input:

[WinEventLog://Security]
disabled = 0
current_only = 1
index = wineventlog
renderXml=true

0 Karma

jkat54
SplunkTrust
SplunkTrust

Valid answers might include McAfee antivirus, custom GPOs, UAC, standard windows auditing when access is granted, custom software, etc.

0 Karma

lukasz92
Communicator

This is only WinEventLog. I catch the events with EventCode 4688.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Yeah doesnt make much sense unless there is some data corruption along the way... packet loss, pinched network cable, disk corruption, etc.

0 Karma

MuS
Legend

Hi lukasz92,

check the CHARSET = <string> in props.conf for this sourcetype on the universal forwarder. Maybe you need to adjust this in your case http://docs.splunk.com/Documentation/Splunk/6.4.0/Admin/Propsconf#GLOBAL_SETTINGS

cheers, MuS

lukasz92
Communicator

Changed also on forwarders. It didn't help.

0 Karma

lukasz92
Communicator

I changed it to CP1252, it didn't help - strange characters appear.

  • I observed that number of strange characters usually equals number of normal characters that should be in this place
0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...

Data Persistence in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. What happens if the OpenTelemetry collector ...

Thanks for the Memories! Splunk University, .conf25, and our Community

Thank you to everyone in the Splunk Community who joined us for .conf25, which kicked off with our iconic ...