base64 decoding in search

[tehrhart]

We're extracting a field from our logs that is base64 encoded and want to display it in its decoded form when searching and reporting. There doesn't appear to be any base64 functionality built-into 'eval'. Does anyone have a solution for base64 decoding on the fly?

[robertosegantin]

Amazing!!!!
Macro works well also on Splunk 6.6.1

[cwayres]

Running with this concept to decode base64 without an app

---

Create lookup used as a converstion matrix with fields: ascii base64bin base64char bin dec hex

| makeresults 
| fields - _time 
| eval bin="0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111" 
| makemv delim=" " bin 
| mvexpand bin 
| map 
    [| makeresults 
    | fields - _time 
    | eval bin="$bin$0000 $bin$0001 $bin$0010 $bin$0011 $bin$0100 $bin$0101 $bin$0110 $bin$0111 $bin$1000 $bin$1001 $bin$1010 $bin$1011 $bin$1100 $bin$1101 $bin$1110 $bin$1111" 
    | makemv delim=" " bin 
    | mvexpand bin ] maxsearches=16 
| mvcombine bin 
| eval dec=mvrange(0,256) 
| eval data=mvzip(bin,dec) 
| fields - bin,dec 
| mvexpand data 
| rex field=data "(?<bin>\d+),(?<dec>\d+)" 
| fields - data 
| eval ascii=printf("%c",dec), hex=printf("%02X",dec) 
| join type=outer dec 
    [ makeresults 
    | fields - _time 
    | eval base64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" 
    | rex field=base64 mode=sed "s/./& /g" 
    | makemv delim=" " base64 
    | eval dec=mvrange(0,64) 
    | eval data=mvzip(base64,dec) 
    | fields - base64,dec 
    | mvexpand data 
    | rex field=data "(?<base64char>[^,]+),(?<dec>[^,]+)" 
    | fields - data ] 
| eval base64bin=if(isnotnull(base64char),substr(bin,3,6),NULL()) 
| append 
    [| makeresults 
    | eval base64bin="000000" 
    | eval base64char="=" 
    | fields - _time ] 
| outputlookup converstionmatrix.csv

Coded the above table. It is way more useful than just for this example

_* Note a duplicate base64bin for 000000 = "=" is added for decoding

Create Macro to Decode base64dec(1): arg1 will be your arguments

 eval b64x_split=split($arg1$,"") 
| lookup converstionmatrix.csv base64char as b64x_split OUTPUT base64bin as b64x_bin 
| eval b64x_join=mvjoin(b64x_bin,"") 
| rex field=b64x_join "(?<b64x_by8>.{8})" max_match=0 
| lookup converstionmatrix.csv bin as b64x_by8 output ascii as b64x_out 
| eval $arg1$_ascii=mvjoin(b64x_out,"")
| fields - b64x_*

Create Macro to Encode base64enc(1): arg1 will be your arguments

eval b64x_split=split($arg1$,"") 
| lookup converstionmatrix.csv ascii as b64x_split output bin as b64x_bin 
| eval b64x_join=mvjoin(b64x_bin,""),b64x_join=if(len(b64x_join)%6>0,b64x_join."000000",b64x_join) 
| rex field=b64x_join "(?<b64x_by6>.{6})" max_match=0 
| lookup converstionmatrix.csv base64bin as b64x_by6 output base64char as b64x_out
| eval $arg1$_base64=mvjoin(b64x_out,"")
| fields - b64x_*

Usage:

| makeresults | eval cs1="MTAxMDEwMTAxCg==~VGhpcyBpcyBhbm90aGVyCg==" | makemv delim=~ cs1 | mvexpand cs1 | `base64dec(cs1)`
| makeresults | eval cs1="splunk" | `base64enc(cs1)` | `base64dec(cs1_base64)`

Christopher Ayres 2019

[malvidin]

The output of the split() command and the printf() command do not always match, so the conversion matrix lookup generated with the command that @cwayres provided does not always work. 

For example:

| eval a1 = printf("%c", 226)
| eval a2 = "â"  `comment("https://www.fileformat.info/info/unicode/char/00e2/index.htm")
| eval s1 = split(a1, "")
| eval s2 = split(a2, "")
| eval matches = if(a1=a2, "true", "false")

 In the output, you can see that both a1 and a2 are both split into two characters, the two bytes that encode that character in UTF8. However, this totally breaks a byte-wise lookup for a base64 function.

This example shows that the conversion matrix that is generated does not create ASCII bytes, it creates Unicode characters stored as UTF8. We get lucky on the first 128 characters, since UTF8 stores those values in one byte that overlaps with ASCII.

| inputlookup conversionmatrix.csv 
| eval is_ascii = if(mvcount(split(ascii, "")) > 1, "false", "true")

[malvidin]

I made a lookup CSV that contained the correct binary values, but Splunk rejected it. when I attempted to upload it. 

"File is binary or file encoding is not supported, only UTF-8 encoded files are supported.

Looks like the hack above is as good as it gets without using the Python Base64 App or the Perl Base64 App. 

[AVOLLMER]

If you are like me and can't install apps, I created this macro to convert base64.

This is the query for my macro:

appendpipe [|dedup $arg1$ |eval converts=split($arg1$, "") |mvexpand converts |lookup base64conversion.csv index as converts OUTPUT value as base64bin |stats list(base64bin) as base64bin by $arg1$| nomv base64bin |rex field=base64bin mode=sed "s/\ //g" |rex field=base64bin "(?<asciibin>.{8})" max_match=100 |mvexpand asciibin| lookup base64conversion.csv index as asciibin output value as ascii | stats list(ascii) as ascii by $arg1$| nomv ascii |rex field=ascii mode=sed "s/\ //g"| table $arg1$ ascii] | selfjoin $arg1$| sort - $arg1$ ascii | filldown ascii

You just pass in a field with the values you want to convert. I have $arg1$ setup as my input argument. You should dedup your results first as this macro uses a selfjoin to attach the converted values back to the input search results. I couldn't find anywhere on the internet where someone had done this before, so it took quite a bit of work to figure all this out.

I don't have enough Karma points to attach a file, so here is the two column csv file I made , which I saved as bas64conversion.csv
index,value,,,,
=,000000,,,,
A,000000,,,,
B,000001,,,,
C,000010,,,,
D,000011,,,,
E,000100,,,,
F,000101,,,,
G,000110,,,,
H,000111,,,,
I,001000,,,,
J,001001,,,,
K,001010,,,,
L,001011,,,,
M,001100,,,,
N,001101,,,,
O,001110,,,,
P,001111,,,,
Q,010000,,,,
R,010001,,,,
S,010010,,,,
T,010011,,,,
U,010100,,,,
V,010101,,,,
W,010110,,,,
X,010111,,,,
Y,011000,,,,
Z,011001,,,,
a,011010,,,,
b,011011,,,,
c,011100,,,,
d,011101,,,,
e,011110,,,,
f,011111,,,,
g,100000,,,,
h,100001,,,,
i,100010,,,,
j,100011,,,,
k,100100,,,,
l,100101,,,,
m,100110,,,,
n,100111,,,,
o,101000,,,,
p,101001,,,,
q,101010,,,,
r,101011,,,,
s,101100,,,,
t,101101,,,,
u,101110,,,,
v,101111,,,,
w,110000,,,,
x,110001,,,,
y,110010,,,,
z,110011,,,,
0,110100,,,,
1,110101,,,,
2,110110,,,,
3,110111,,,,
4,111000,,,,
5,111001,,,,
6,111010,,,,
7,111011,,,,
8,111100,,,,
9,111101,,,,
+,111110,,,,
/,111111,,,,
00100001,!,,,,
00100010,"""",,,,
00100011,#,,,,
00100100,$,,,,
00100101,%,,,,
00100110,&,,,,
00100111,',,,,
00101000,(,,,,
00101001,),,,,
00101010,*,,,,
00101011,+,,,,
00101100,",",,,,
00101101,-,,,,
00101110,.,,,,
00101111,/,,,,
00110000,0,,,,
00110001,1,,,,
00110010,2,,,,
00110011,3,,,,
00110100,4,,,,
00110101,5,,,,
00110110,6,,,,
00110111,7,,,,
00111000,8,,,,
00111001,9,,,,
00111010,:,,,,
00111011,;,,,,
00111100,<,,,,
00111101,=,,,,
00111110,>,,,,
00111111,?,,,,
01000000,@,,,,
01000001,A,,,,
01000010,B,,,,
01000011,C,,,,
01000100,D,,,,
01000101,E,,,,
01000110,F,,,,
01000111,G,,,,
01001000,H,,,,
01001001,I,,,,
01001010,J,,,,
01001011,K,,,,
01001100,L,,,,
01001101,M,,,,
01001110,N,,,,
01001111,O,,,,
01010000,P,,,,
01010001,Q,,,,
01010010,R,,,,
01010011,S,,,,
01010100,T,,,,
01010101,U,,,,
01010110,V,,,,
01010111,W,,,,
01011000,X,,,,
01011001,Y,,,,
01011010,Z,,,,
01011011,[,,,,
01011100,\,,,,
01011101,],,,,
01011110,^,,,,
01011111,_,,,,
01100000,@,,,,
01100001,a,,,,
01100010,b,,,,
01100011,c,,,,
01100100,d,,,,
01100101,e,,,,
01100110,f,,,,
01100111,g,,,,
01101000,h,,,,
01101001,i,,,,
01101010,j,,,,
01101011,k,,,,
01101100,l,,,,
01101101,m,,,,
01101110,n,,,,
01101111,o,,,,
01110000,p,,,,
01110001,q,,,,
01110010,r,,,,
01110011,s,,,,
01110100,t,,,,
01110101,u,,,,
01110110,v,,,,
01110111,w,,,,
01111000,x,,,,
01111001,y,,,,
01111010,z,,,,,

[valiquet]

I fixed the pipe put still can`t fix the spaces! Thanks

|,01111100,,,,,
01111100,|,,,,,

[tpaulsen]

Awesome! Works very well! Even with Splunk 5.0.5.

[dwaddle · Accepted solution]

I just posted an app that has a base64 custom command packaged in it. See if this does what you need.

[neogeek83]

Anyone using this with Splunk6, does it still work?

[acurcio_splunk]

You may want to try the new app; "Base64"
https://splunkbase.splunk.com/app/1922/