topic Detecting different character sets in an email subject in Splunk Dev

Detecting different character sets in an email subject

sheamus69 — Thu, 31 Jan 2019 15:32:55 GMT

I want to report on emails containing subjects that are using difference character sets, such as Chinese, Russian, Greek alphabet, etc.

Is there an easy way to pull out the character encoding from the emails?

eg:

Sender: someone@somewhere.com
Sender: 你好，世界

Re: Detecting different character sets in an email subject

dkeck — Thu, 31 Jan 2019 16:16:06 GMT

HI,

its a bit bulky, but wouldn´t it be working if you use your regex to find everything except chracters you have in your charset?

[^:.@,\s+\w+]

This is matching the chinese characters

And at least for chinese there is a method to match all chracters with \p{Han}. Seems to work in splunk.

| makeresults | eval aaa="世界" | rex field=aaa (?<my_aaa>\p{Han}.*)

Re: Detecting different character sets in an email subject

sheamus69 — Thu, 31 Jan 2019 17:12:19 GMT

That was an interesting approach I hadn't considered. The problem I am finding is that there seem to be lots of edge cases, such as é, í, !, ", £, $, %, ... I keep going back and finding more to exclude.

Not sure if this is the best way to do it, or if theres an alternate approach?

Re: Detecting different character sets in an email subject

dkeck — Thu, 31 Jan 2019 17:25:11 GMT

I found a List of all languages here :

https://www.regular-expressions.info/unicode.html#prop scroll down to Unicode Scripts and Unicode Blocks.

You could use [^\p{Latin}], since everything you are looking for is non latin?!

I think thats the closest you can get, by using the rex above

Re: Detecting different character sets in an email subject

dkeck — Fri, 01 Feb 2019 05:22:22 GMT

Would be great if you could accept and upvote the question, thank you 🙂