[HEC] SHIFT-JIS Character recognition

[sylim_splunk]

I have some data that is being collected from an AWS lambda and delivered to Splunk via HEC with the listeners on the indexers.

This data contains Japanese characters but is not displaying properly in SplunkWeb.

I have applied a host level stanza on both the search head and indexers to CHARSET = SHIFT-JIS, however, the data is still displayed as question marks in SplunkWeb. I have tried AUTO, UTF-8 and SHIFT-JIS without success.

[sylim_splunk · Accepted solution]

When you send data over HEC, especially for international character data, make sure to specify the charset encoding for the data set. That way the receiving end knows its charset encoding and how to decrypt. Oftentimes the encryption method is different than what you expect.

Here's a good example for your reference:

https://medium.com/@rysartem/sending-data-to-splunk-hec-in-a-right-way-4a84af3c44e2