topic Re: find events that contain non english words in Splunk Search

How can I find events that contain non english words?

indeed_2000 — Tue, 21 Jun 2022 15:44:40 GMT

how can I find events that contain non english words?

e.g i have log file that some lines contain germany or arabic words, how can i recognize these lines?

thanks

Re: find events that contain non english words

DanielMustaine — Tue, 21 Jun 2022 03:35:30 GMT

I can't help you too much for non-English words in ASCII character sets, but for languages with characters in the unicode set you could consider using Unicode Categories:

https://www.regular-expressions.info/unicode.html

Quick example:

| makeresults | eval text="كلب means dog according to google" | rex field=text "(?<capturetext>\p{Arabic}*)" | table text capturetext

Re: find events that contain non english words

VatsalJagani — Tue, 21 Jun 2022 06:28:19 GMT

@indeed_2000 - Try the below search, it will find anything that contains anything other than desired character set.

<your base search> | rex _raw="[^\x00-\x7F]"

I hope this helps!!!

Re: find events that contain non english words

indeed_2000 — Tue, 21 Jun 2022 14:58:29 GMT

@DanielMustaine not work on this

2022-06-20 11:16:10,381 INFO [APP] log in : 38773763@#123@دالكي

any idea?

Re: find events that contain non english words

indeed_2000 — Tue, 21 Jun 2022 15:03:40 GMT

@VatsalJagani not work!

Re: find events that contain non english words

VatsalJagani — Tue, 21 Jun 2022 15:35:41 GMT

@indeed_2000 - The regex does work in the event that you provided.

https://regex101.com/r/sG3IdX/1

Re: find events that contain non english words

indeed_2000 — Tue, 21 Jun 2022 16:01:01 GMT

Consider each character as separate, in this line i only have one non English word!

Excpected output:

| table NonEnglish

دالكي

Any idea?

Re: find events that contain non english words

DanielMustaine — Thu, 23 Jun 2022 00:47:25 GMT

Hey, try this:

| makeresults | eval text="2022-06-20 11:16:10,381 INFO [APP] log in : 38773763@#123@دالكي " |rex field=text "(?<capturetext>[\p{Arabic}]+)" | table text capturetext

Alternatively using VatsalJagani's Regex (which will match all non-ASCII characters so get you the Russian sets etc.) like:

| makeresults | eval text="2022-06-20 11:16:10,381 INFO [APP] log in : 38773763@#123@دالكي " |rex field=text "(?<capturetext>[^\x00-\x7F]+)" | table text capturetext

If you know which field the non-standard characters will be in make sure to sub that fieldname into field=xxx. Otherwise, and it will perform poorly, but you can look over the whole log with field=_raw.

Re: find events that contain non english words

indeed_2000 — Thu, 23 Jun 2022 05:30:58 GMT

Still not work as expected, if more that one non english word exist on each line i expect consider them.

e.g.

Expected result for below line: دالكي هلت

"2022-06-20 11:16:10,381 INFO [APP] log in : 38773763@#123@دالكي هلت "

Re: find events that contain non english words

DanielMustaine — Fri, 24 Jun 2022 03:30:41 GMT

Try this:

| rex max_match=0 field=text "(?<capturetext>\p{Arabic}[\p{Arabic}\s]+)"

This one I didn't have as much success with but not much time to play.

| rex field=text max_match=0 "(?<capturetext>[^\x00-\x7F][^\x00-\x7F\s]+)"

Max match will also mean that multiple phrases in one log are pulled into capturetext as new multivalue field values.

Re: find events that contain non english words

indeed_2000 — Sat, 25 Jun 2022 03:38:25 GMT

is there anyway to consider whole non english charecter in each line as one extraction?

2022-06-20 11:16:10,381 INFO [APP] log in : 38773763@#123@دالكي string تست

exception result:

تست دالكي

Re: find events that contain non english words

PickleRick — Sat, 25 Jun 2022 09:29:47 GMT

Just remember that as it was stated before it has nothing to do with language as such. It will not capture, for example, a sentence "moja matka jada pomidory" ("my mother eats tomatoes" in Polish) even though it's clearly not an English sentence.