topic How do I search by unicode value? in Splunk Search

How do I search by unicode value?

yyossef — Thu, 30 Aug 2018 10:49:41 GMT

Hi,

I have the following example record:

30/08/2018 13:30:27.996;VM1;ASH;AccessModule;processPacketBuffer;MSISDN;xxxxxxxxxxxx;;INFO;;;Return Access ; "msisdn":"xxxxxxxxx","Type":"\u0006","APN":"aaa","imsi":"xxxxxxxx","imei":"xxxxxxxxx","SGSN":null,"Remote IP Address":"xx.xx.xx.xx","TotalTimeInMS":0}

I can not search by Type, because it is a unicode value, and Splunk does not parse it correctly.

The are 2 possible Type values: 1. "\u0006" 2. "\u0003".

I am using the following splunk search:
mysearch | spath input=anyparams | search Type="\u0006"

The problem is that i receive no result,

How should I use the search, when the field contains a unicode value?

Thanks in advance,

Yossi

Re: How do I search by unicode value?

niketn — Thu, 30 Aug 2018 14:03:35 GMT

@yyossef, if you are searching unicode stored as text you would need to escape backslash by prefixing another backslash i.e. "\\u0006" or "\\u0003" in your SPL.

Following is and example to use the same in search filter or eval function

 <yourCurrentSearch>
| eval TypeDescription=case(Type=="\\u0006","ACKNOWLEDGE",Type=="\\u0004","END OF TEXT",true(),"Others")
| search Type="\\u0006" OR TypeDescription="ACKNOWLEDGE"

Following is run anywhere search based on sample data provided:

| makeresults
| eval _raw="30/08/2018 13:30:27.996;VM1;ASH;AccessModule;processPacketBuffer;MSISDN;xxxxxxxxxxxx;;INFO;;;Return Access ; \"msisdn\":\"xxxxxxxxx\",\"Type\":\"\\u0006\",\"APN\":\"aaa\",\"imsi\":\"xxxxxxxx\",\"imei\":\"xxxxxxxxx\",\"SGSN\":null,\"Remote IP Address\":\"xx.xx.xx.xx\",\"TotalTimeInMS\":0}"
| extract pairdelim="," kvdelim=":"
| eval TypeDescription=case(Type=="\\u0006","ACKNOWLEDGE",Type=="\\u0004","END OF TEXT",true(),"Others")
| search Type="\\u0006" OR TypeDescription="ACKNOWLEDGE"

Re: How do I search by unicode value?

yyossef — Thu, 30 Aug 2018 14:35:42 GMT

Hi @niketnilay,

Thanks for your prompt response.
Still no luck, the search result is empty.

When using your 4 example, the result came back with only the deafault value "Other", meaning, no match was found.
I am not sure that the unicode is stored as text, i think it is display as text by the system, but stored as unicode value.
Do you have idea how to verify that? or how to search by unicode value?

Re: How do I search by unicode value?

niketn — Thu, 30 Aug 2018 15:02:17 GMT

@yyossef, I am not sure whether the Type field is actually being extracted or not... So first let us try a different approach. Following example does not try to extract Type field. Instead searched for unicode characters in raw data.

| makeresults
| eval _raw="30/08/2018 13:30:27.996;VM1;ASH;AccessModule;processPacketBuffer;MSISDN;xxxxxxxxxxxx;;INFO;;;Return Access ; \"msisdn\":\"xxxxxxxxx\",\"Type\":\"\\u0006\",\"APN\":\"aaa\",\"imsi\":\"xxxxxxxx\",\"imei\":\"xxxxxxxxx\",\"SGSN\":null,\"Remote IP Address\":\"xx.xx.xx.xx\",\"TotalTimeInMS\":0}"
| eval TypeDescription=case(searchmatch("\\u0006"),"ACKNOWLEDGE",searchmatch("\\u0004"),"END OF TEXT",true(),"Others")
| search TypeDescription="ACKNOWLEDGE"

Re: How do I search by unicode value?

ddrillic — Fri, 31 Aug 2018 00:09:43 GMT

Looking at Unicode Character 'ACKNOWLEDGE' (U+0006)

It tells us that \u0006 is not a unicode/utf-8 character representation - it's the way several programming languages chose to represent it.

Re: How do I search by unicode value?

yyossef — Sun, 02 Sep 2018 06:05:46 GMT

Hi @niketnilay,

Yours other suggestion using searchmatch worked.

| makeresults
| eval _raw="30/08/2018 13:30:27.996;VM1;ASH;AccessModule;processPacketBuffer;MSISDN;xxxxxxxxxxxx;;INFO;;;Return Access ; \"msisdn\":\"xxxxxxxxx\",\"Type\":\"\u0006\",\"APN\":\"aaa\",\"imsi\":\"xxxxxxxx\",\"imei\":\"xxxxxxxxx\",\"SGSN\":null,\"Remote IP Address\":\"xx.xx.xx.xx\",\"TotalTimeInMS\":0}"
| eval TypeDescription=case(searchmatch("\u0006"),"ACKNOWLEDGE",searchmatch("\u0004"),"END OF TEXT",true(),"Others")
| search TypeDescription="ACKNOWLEDGE"

Why would searchmatch works while Type=="\u0006" did not?

Re: How do I search by unicode value?

niketn — Sun, 02 Sep 2018 14:31:23 GMT

@yyossef Type field is not getting auomatically extracted as part of Search Time field discovery. The searchmatch command finds the pattern match in the entire raw data. You would need to create your own Field Extraction to create a Type field based on Regular Expression.

I am glad your issue is resolved. Do let us know if you need further help. Do up vote the answer/comments that helped! 🙂

Re: How do I search by unicode value?

MichaelArsenaul — Tue, 21 Jan 2020 19:47:56 GMT

Just curious: why are unicode values not being cleansed/translated before the information gets sent to Stripe? As far as I know, data like this very rarely makes its way into Splunk, and much of what passes as weird UTF-8 codes do not make it into Splunk at all.

Ian Quick shared this example code with us that shows how to test for uTF-8 characters and strip them out: https://github.com/Shopify/shopify-tracing/commit/816ba2aef3c6ee8a232766028181b7b1ca03a2b1

I'd highly recommend cleansing your data before emitting to Stripe. Once the data is in Splunk, 99.9% of the UTF code will be lost and Splunk will not help you debug that issue. Cleansing your output before it hits Stripe is probably the best course of action.