Monitoring Splunk

24284 bytes long Event log message causes splunkd to crash - workaround

FloydATC
Explorer

One of my Exchange 2010 hubcas servers is logging monstrous messages into the Event Log about conflicting updates from two ActiveSync devices accessing the same mailbox. (Each event Message is 24284 bytes and contains several debug traces and contains pretty much everything except what that particular Microsoft employee had for breakfast that day)

Anyway, when I try to select that Message into a table, the splunk server craps itself and must be manually restarted:
... | table _time, host, LogName, Message

Tags (4)
0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

Also, I suspect that the Message field has a certain format. View the event in the event viewer and then extract just the information you need and display that. I suspect the Device ID, Device Type and username are logged in there. The actual processing doesn't care what size the event is - only the display.

0 Karma

FloydATC
Explorer

Yes, the problem only seems to affect 'table' view of the 'Message' field, showing the '_raw' event works as expected.

0 Karma

FloydATC
Explorer

Workaround: Limit the potentially huge string to a more sane length, say 1024 characters

... | eval msg=substr(Message,0,1024) | table _time, host, LogName, msg

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...