TA-mailclient handling foreign characters

[LIUJIEER]

This TA worked fine with emails in English. However, it does not work for foreign characters such as Chinese, Japanese. I received this debugging message as shown below and it stopped indexing once there is a email with foreign characters in the inbox, unless i annually remove it.

ERROR ExecProcessor - message from "python /apps/splunk/etc/apps/TA-mailclient/bin/mail.py" ERRORunknown encoding: iso-2022-jp

Is there any way i can index emails with foreign characters with this TA. Alternatively, can i just drop those emails with any foreign characters?

Thank you

[alexstackharbor]

Hi Seun,

Will test it out and let you know how it goes.

Thanks for your efforts.

Regards.

[alexstackharbor]

Hi Seun,

Unfortunately I am not able to get the *.dev versions to work properly for me, I get this error:
Mail found with unexpected codec -

Only the headers get indexed and none of the content, this is happening for ALL mail, even the ones that were working properly before.

Do you have an idea where I should start looking to resolve this issue?

Thank you,
Regards.

[seunomosowon]

I’ll fix the test for the codecs this week.

[seunomosowon]

Hi Alex,

It's something to do with the codecs not being supported in the python that comes with Splunk. It would actually be better if your channel mail settings were UTF8. Not sure yet if i can find a way to add additional character sets to this. But i'll let you know following some research.

Cheers,

[alexstackharbor]

Hi Seun,

The strange thing is, the previous versions work correctly ( for all but a certain few foreign chars)

After updating to the *dev version, It starts throwing this error on mails that it was properly indexing before and almost nothing makes it into the index anymore.

Using 1.3.5 (which seems to not be available anymore) it seems to work properly, however 1.3.5dev, 1.3.6dev, and 1.3.7dev all throw the same errors for mail that 1.3.5 indexes without issues.

This one really has me scratching my head, as I am sure there is something simple/small that I/we are missing.

In any case I really appreciate your efforts, let me know if you have any ideas on where to look to solve this issue.

Regards.

[seunomosowon]

Added one more version. 1.3.5dev

It should not stop upon reaching that mails encoded using an unsupported codec. Please give it a try.

Cheers

[alexstackharbor]

Hi Seun, if it’s set to delete the emails upon indexing what would be the expected behaviour when it comes across something it can’t index? Will it discard the email and continue to index the others or will it leave it behind and continue processing the rest and deleting only after being indexed??

[LIUJIEER]

You mean I can do something on my Splunk/system settings to support these encodings?