topic Is there any way to decode an encoded html values saved in a log file? in Splunk Search

Is there any way to decode an encoded html values saved in a log file?

Boopalan — Fri, 05 Apr 2019 09:41:58 GMT

I want decode all the encoded html values present in an log file while indexing itself.
Is there any way to do it ?

Re: Is there any way to decode an encoded html values saved in a log file?

Boopalan — Mon, 08 Apr 2019 09:58:29 GMT

Hey Splunk folks,
Is there any possible way/ideas to do that?

Re: Is there any way to decode an encoded html values saved in a log file?

niketn — Mon, 08 Apr 2019 12:57:24 GMT

@Boopalan any example from the log where html values are encoded? Which kind of encoding is in place? Also are these specific technology logs or are they custom logs? Please mock/anonymize any sensitive data.

Re: Is there any way to decode an encoded html values saved in a log file?

Boopalan — Mon, 08 Apr 2019 13:50:36 GMT

@niketnilay, PFB sample of encoded html values in the log file.

%[datetime] [Default: 0] [] [INFO ] [*] - &#xa;--  Start of getter Meods&#xa;retiremevice.io.ReturnCd&#x3a; 0&#xa;retiremevice.io.RtSCd&#x3a;

Re: Is there any way to decode an encoded html values saved in a log file?

pkeenan87 — Mon, 08 Apr 2019 14:08:30 GMT

You could try and decode it at search time with the urldecode eval function:

https://docs.splunk.com/Documentation/Splunk/7.2.5/SearchReference/TextFunctions

Re: Is there any way to decode an encoded html values saved in a log file?

Boopalan — Mon, 08 Apr 2019 14:20:46 GMT

@pkeenan87, urldecode function is not working as expected. I tried doing that but that is working only for decoding values of url addresses not for an string containing ASCII encoded values in html.

Re: Is there any way to decode an encoded html values saved in a log file?

niketn — Mon, 08 Apr 2019 14:55:50 GMT

@Boopalan, for the above text what is the expected decoded characters? I am not sure if this looks like either encoding or escaping of special characters. Would need to wait for others to comment. Or more details would help us assist you better!

Re: Is there any way to decode an encoded html values saved in a log file?

dmarling — Mon, 08 Apr 2019 15:55:18 GMT

Urldecode decodes with the url encoding that starts with a precentage sign. You can manipulate the data a bit to force it to work. Using the most recent example by @Boopalan I got it to work, but I had to manually account for the HTML Encoded Line Feed character

Run anywhere example:

| makeresults count=1
| eval data="- &#xa;--  Start of getter Meods&#xa;retiremevice.io.ReturnCd&#x3a; 0&#xa;retiremevice.io.RtSCd&#x3a;"
| eval test=data
| rex mode=sed field=test "s/&#xa;/\n/g"
| rex mode=sed field=test "s/&#x?([^\;]+);/%\1/g"
| eval test2=urldecode(test)

It makes it look like this when it runs:

- 
--  Start of getter Meods
retiremevice.io.ReturnCd: 0
retiremevice.io.RtSCd:

Re: Is there any way to decode an encoded html values saved in a log file?

sampathramtvnr — Tue, 17 Dec 2019 09:51:05 GMT

urldecode works for decoding the values of URL addresses and strings
Try the below examples:

with url :
| makeresults

| eval field1= "f%23has%2Bofh%20a"
| eval field1 = urldecode(field1)

with string:
For example you already have field value then,
| rex mode=sed field=field1 "s/ / /g"
| eval a=urldecode(field1)