Getting Data In

malformedEventIndex, how to troubleshoot and fix logs ending up here

kmfpo
Explorer

Hello all,

I created a malformedEventIndex ( malformedevent), per inputs.conf. I see 400 million+/day from thousands of hosts going to this index from my syslog servers (have a HF that sends to indexer cluster).

I tried looking at the events to see what would cause them to end up in this index, and patterns shows:

100%    __default_indexprocessor_body 

This doesn't tell me anything. I went through the reasons as to why events may end up here and none seem to match.

* Routes the following kinds of events to the specified index:
    * events destined for read-only indexes ### we don't have these
    * log events destined for datatype=metric indexes ### no logs on syslog server go to metric indexes
    * log events with invalid raw data values, like all-whitespace raw ### I cat log files on syslog server and they are not all-whitespace 
    * metric events destined for datatype=event indexes ### these systems are not sending metric events
    * metric events with invalid metric values, like non-numeric values ### see above
    * metric events lacking required attributes, like metric name ### see above

Documentation on this index is extremely sparse so I am not sure where to go from here. Please help.

0 Karma

archive
New Member

I am having the same issue, would love to bump this for visibility and for an update. Thanks!

0 Karma

GregoryHoward
New Member

Hi,

We made a support ticket about this behavior.

This was the answer :

"

Reproduction went quickly this time, I was able to successfully reproduce this issue on 8.0.1 and 8.1.2 versions.

This smells like a potential bug impacting many versions.

I am raising now an internal request to developers team to check and review it.

I will keep you informed about the progress, but it may take some time to get developers engaged.

"

Regards,

Gregory

0 Karma

kyaparla
Path Finder

Hi All,

Any update on this issue?  We were seeing this issue on 8.1.3 version, and continue to see after upgrading to 8.2.2.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...