malformedEventIndex, how to troubleshoot and fix logs ending up here

[kmfpo]

Hello all,

I created a malformedEventIndex ( malformedevent), per inputs.conf. I see 400 million+/day from thousands of hosts going to this index from my syslog servers (have a HF that sends to indexer cluster).

I tried looking at the events to see what would cause them to end up in this index, and patterns shows:

100%    __default_indexprocessor_body 

This doesn't tell me anything. I went through the reasons as to why events may end up here and none seem to match.

* Routes the following kinds of events to the specified index:
    * events destined for read-only indexes ### we don't have these
    * log events destined for datatype=metric indexes ### no logs on syslog server go to metric indexes
    * log events with invalid raw data values, like all-whitespace raw ### I cat log files on syslog server and they are not all-whitespace 
    * metric events destined for datatype=event indexes ### these systems are not sending metric events
    * metric events with invalid metric values, like non-numeric values ### see above
    * metric events lacking required attributes, like metric name ### see above

Documentation on this index is extremely sparse so I am not sure where to go from here. Please help.

[archive]

I am having the same issue, would love to bump this for visibility and for an update. Thanks!

[GregoryHoward]

Hi,

We made a support ticket about this behavior.

This was the answer :

"

Reproduction went quickly this time, I was able to successfully reproduce this issue on 8.0.1 and 8.1.2 versions.

This smells like a potential bug impacting many versions.

I am raising now an internal request to developers team to check and review it.

I will keep you informed about the progress, but it may take some time to get developers engaged.

"

Regards,

Gregory

[kyaparla]

Hi All,

Any update on this issue?  We were seeing this issue on 8.1.3 version, and continue to see after upgrading to 8.2.2.