Developing for Splunk Enterprise

Detecting different character sets in an email subject

sheamus69
Communicator

I want to report on emails containing subjects that are using difference character sets, such as Chinese, Russian, Greek alphabet, etc.

Is there an easy way to pull out the character encoding from the emails?

eg:

Sender: someone@somewhere.com
Sender: 你好,世界
Tags (1)
0 Karma
1 Solution

dkeck
Influencer

I found a List of all languages here :

https://www.regular-expressions.info/unicode.html#prop scroll down to Unicode Scripts and Unicode Blocks.

You could use [^\p{Latin}], since everything you are looking for is non latin?!

I think thats the closest you can get, by using the rex above

View solution in original post

dkeck
Influencer

I found a List of all languages here :

https://www.regular-expressions.info/unicode.html#prop scroll down to Unicode Scripts and Unicode Blocks.

You could use [^\p{Latin}], since everything you are looking for is non latin?!

I think thats the closest you can get, by using the rex above

dkeck
Influencer

Would be great if you could accept and upvote the question, thank you 🙂

0 Karma

dkeck
Influencer

HI,

its a bit bulky, but wouldn´t it be working if you use your regex to find everything except chracters you have in your charset?

like

[^:.@,\s+\w+]

This is matching the chinese characters

OR

And at least for chinese there is a method to match all chracters with \p{Han}. Seems to work in splunk.

| makeresults | eval aaa="世界" | rex field=aaa (?<my_aaa>\p{Han}.*)
0 Karma

sheamus69
Communicator

That was an interesting approach I hadn't considered. The problem I am finding is that there seem to be lots of edge cases, such as é, í, !, ", £, $, %, ... I keep going back and finding more to exclude.

Not sure if this is the best way to do it, or if theres an alternate approach?

0 Karma