Conver to Hexadecimal code to bit value

[smanojkumar]

Hello,
   I would like to convert my hexadecimal code to a bit value based on this calculation. 

Hex code - 0002

Seperate 2 bytes each  00/02

2 Byte bitmask
Byte 0: HEX = 00 - 0000 0000
Byte 1: HEX = 02 - 0000 0010

Byte 1 Byte0 - 0000 0010 0000 0000

calculate the non zero th position values from right side

Byte combination  - 0000 0010 0000 0000
Position -                      9 8765  4321

At position 10, we got 1 while counting from right side. so the bit value is 9.


I need to calculate this in splunk, where the HEX_Code is the value from the lookup.

Thanks in Advance!
Happy Splunking!

[tscroggins]

Hi @smanojkumar,

This is a 16-bit adaptation of Gaudet's algorithm from Hacker's Delight Second Edition (Warren, 2013):

| makeresults
| eval HEX_Code="0002"
``` convert to number ```
| eval x=tonumber(HEX_Code, 16)
``` swap bytes ```
| eval x=bit_shift_right(x, 8)+bit_and(bit_shift_left(x, 8), 65280)
``` calculate number of trailing zeros (ntz) ```
| eval y=bit_and(x, 65535-x+1)
| eval bz=if(y>0, 0, 1), b3=if(bit_and(y, 255)>0, 0, 8), b2=if(bit_and(y, 3855)>0, 0, 4), b1=if(bit_and(y, 13107)>0, 0, 2), b0=if(bit_and(y, 21845)>0, 0, 1)
| eval ntz=bz+b3+b2+b1+b0
``` ntz=9 ```

[smanojkumar]

Hello @tscroggins ,
   Thanks for your reply!

   I'm having this error - "Error in 'EvalCommand': The 'bit_shift_right' function is unsupported or undefined."

Can you help in resolving this error.

Thanks in Advance!

[tscroggins]

Here's an alternative that uses a few helper macros to replace the bitwise eval functions. Bit rotate functions would be a nice addition to Splunk, as would a parameter on all bitwise functions to specify width.

| makeresults
| eval HEX_Code="0002"
``` convert to number ```
| eval x=tonumber(HEX_Code, 16)
``` swap bytes ```
| eval t=`bitshl(x, 8)`, x=`bitshr(x, 8)`+`bitand_16(t, 65280)`
``` calculate number of trailing zeros (ntz) ```
| eval t=65535-x+1, y=`bitand_16(x, t)`
| eval bz=if(y>0, 0, 1), b3=if(`bitand_16(y, 255)`>0, 0, 8), b2=if(`bitand_16(y, 3855)`>0, 0, 4), b1=if(`bitand_16(y, 13107)`>0, 0, 2), b0=if(`bitand_16(y, 21845)`>0, 0, 1)
| eval ntz=bz+b3+b2+b1+b0
``` ntz=9 ```

# macros.conf

[bitand_16(2)]
args = x, y
definition = sum(1 * (floor($x$ / 1) % 2) * (floor($y$ / 1) % 2), 2 * (floor($x$ / 2) % 2) * (floor($y$ / 2) % 2), 4 * (floor($x$ / 4) % 2) * (floor($y$ / 4) % 2), 8 * (floor($x$ / 😎 % 2) * (floor($y$ / 😎 % 2), 16 * (floor($x$ / 16) % 2) * (floor($y$ / 16) % 2), 32 * (floor($x$ / 32) % 2) * (floor($y$ / 32) % 2), 64 * (floor($x$ / 64) % 2) * (floor($y$ / 64) % 2), 128 * (floor($x$ / 128) % 2) * (floor($y$ / 128) % 2), 256 * (floor($x$ / 256) % 2) * (floor($y$ / 256) % 2), 512 * (floor($x$ / 512) % 2) * (floor($y$ / 512) % 2), 1024 * (floor($x$ / 1024) % 2) * (floor($y$ / 1024) % 2), 2048 * (floor($x$ / 2048) % 2) * (floor($y$ / 2048) % 2), 4096 * (floor($x$ / 4096) % 2) * (floor($y$ / 4096) % 2), 8192 * (floor($x$ / 8192) % 2) * (floor($y$ / 8192) % 2), 16384 * (floor($x$ / 16384) % 2) * (floor($y$ / 16384) % 2), 32768 * (floor($x$ / 32768) % 2) * (floor($y$ / 32768) % 2))
iseval = 0

[bitshl(2)]
args = x, k
definition = floor(pow(2, $k$) * $x$)
iseval = 0

[bitshr(2)]
args = x, k
definition = floor(pow(2, -$k$) * $x$)
iseval = 0

[ITWhisperer]

bit-wise functions only came into Splunk Enterprise in 9.2.0 and in 9.1 in Cloud Services (according to the documentation) - which version of Splunk are you using?

[yuanliu]

@smanojkumar Can you confirm that results you are looking for are like the following?

hex	padded_binary	nonzero_bits
0002	00000010 00000000	9
00200100	00000000 00000001 00100000 00000000	13 16
01100011	00010001 00000000 00010000 00000001	0 12 24 28

This sounds like some data compression game.  I can't think of a practical reason to do this in SPL.  Is this some sort of homework?

Anyway, here is a more or less literal way to interpret your instructions:

| eval idx = mvrange(0, len(hex) / 2)
| eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2)))
| eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary")))
| eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary)))
| eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1
| foreach reverse_bits mode=multivalue
    [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))]
| fields hex padded_binary nonzero_bits

Note mvreverse on padded binary is sort of expensive and can be avoided by arithmetics if there are lots of data.

Here is the emulation of the three examples you give:

| makeresults format=csv data="hex
0002
00200100
01100011"
``` data emulation above ```

Apply the algorithm to this emulation gives the results tabulated at the top.

[PickleRick]

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by @smanojkumar 's explanation as to how the algorithm is supposed to work. Does it work on 16-bit integers only? Does it work on any length stream of data? Does it always produce 32-bit integers? Or does the result grow with the length of the argument? It's so badly specified...

[yuanliu · Accepted solution]

---

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by 

---

@PickleRick Lol you wouldn't believe how much time I spent trying to decipher the OP's intent from the various posted replies to everybody's attempt to help.  After hours of scrolling up and down, back and forth, I distilled the instructions into the following algorithm given an even numbered HEX string, e.g., aabbcc

1. Break the string into 2-HEX chunks. (OP used the term 2-bytes - I realize that is actually 4-bytes)
2. Convert each chunk into binary.
3. Reverse the order of the binary chunks.
4. Count the positions of nonzero bits of the full reversed binary string from the right.

(As I said, I can't think of a practical purpose of this exercise. By the way, to anyone who is going to ask a question here, even though I strongly encourage describing problem without SPL first, please make the description as algorithmic as possible.)  As a weird game, this applies to any even-length HEX string.  Here's a sequence of up to 16 HEX characters.

hex	padded_binary	nonzero_bits
01	00000001	0
0002	00000010 00000000	9
000003	00000011 00000000 00000000	16 17
00000004	00000100 00000000 00000000 00000000	26
0000000005	00000101 00000000 00000000 00000000 00000000	32 34
000000000006	00000110 00000000 00000000 00000000 00000000 00000000	41 42
00000000000007	00000111 00000000 00000000 00000000 00000000 00000000 00000000	48 49 50
0000000000000008	00001000 00000000 00000000 00000000 00000000 00000000 00000000 00000000	59
90000000000000	00000000 00000000 00000000 00000000 00000000 00000000 10010000	4 7
a00000000000	00000000 00000000 00000000 00000000 00000000 10100000	5 7
b000000000	00000000 00000000 00000000 00000000 10110000	4 5 7
c0000000	00000000 00000000 00000000 11000000	6 7
d00000	00000000 00000000 11010000	4 6 7
e000	00000000 11100000	5 6 7
f0	11110000	4 5 6 7

Another thing I realize is that I must handle 2-HEX (single-chunk) specially.  Here is the emulation code

| makeresults format=csv data="hex
01
0002
000003
00000004
0000000005
000000000006
00000000000007
0000000000000008
90000000000000
a00000000000
b000000000
c0000000
d00000
e000
f0"
``` data emulation above ```
| eval idx = mvrange(0, len(hex) / 2)
| eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2)))
| eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary")))
| eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary)))
| eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1
| foreach reverse_bits mode=multivalue
    [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))]
| fields hex padded_binary nonzero_bits

(Technically this works for odd number of HEX characters, too, if OP can define where to split.) 

[smanojkumar]

Hi @yuanliu & @ITWhisperer  & @tscroggins  & @PickleRick  & @dural_yyz ,

    Thanks everyone for your time, it works for me.

Thanks in Advance!

[smanojkumar]

Hello @yuanliu ,
   Thanks for your response!

    I'm having this error "Error in 'EvalCommand': The arguments to the 'tostring' function are invalid.", can you please help me in this.


Thanks in advance!

[ITWhisperer]

"binary" only came into Splunk Enterprise in 9.2.0 and doesn't appear to be in Cloud Services yet (according to the documentation)

[smanojkumar]

Hello @ITWhisperer ,
   Thanks for that.
   I'm currebtly using Splunk Enterprise with Version 9.1.1, So thats the reason.

   Any alternative way to work on this with this version?

Thanks for pointing it out.

regards,
Manoj Kumar S

[ITWhisperer]

Having converted the number to hex, perform 16 replacements, starting with 0, then 1, replacing the hex digit with the corresponding binary equivalent.

[smanojkumar]

Hello @ITWhisperer ,

   Thanks for your resposne!

   If you don't mind changing the code as well.

Thansk a lot for your resposne!

[ITWhisperer]

Here are the first four, I am sure you can workout from this how to do the other 12 hex digits

| eval ASbinary=if(idx < 1, replace(replace(replace(replace(reverse2hex,"0","0000"),"1","0001"),"2","0010"),"3","0011"), mvmap(reverse2hex, replace(replace(replace(replace(reverse2hex,"0","0000"),"1","0001"),"2","0010"),"3","0011")))

[smanojkumar]

Hi @ITWhisperer ,

   Thanks, It works on that place.

[PickleRick]

Nice job. I'm stil not convinced it's how it was supposed to work 😉

[smanojkumar]

Another example

Hex code - 00200100

Seperate 2 bytes each  00/20/01/00

4 Byte bitmask
Byte 0: HEX = 00 -  0000 0000
Byte 1: HEX = 20 - 0010 0000
Byte 2: HEX = 01 - 0000 0001
Byte 3: HEX = 00 - 0000 0000

Byte 3 Byte 2 Byte1 Byte 0 - 0000 0000 0000 0001 0010 0000 0000 0000

calculate the non zero th position values from right side

Byte combination  - 0000 0000 0000 0001 0010 0000 0000 0000
Position -                                                               16       13        ...9 8765  4321

At position 14 and 17, we got 1 while counting from right side. so the bit value is 13 and 16.

[PickleRick]

OK. This is confusing. You have four hexadecimal digits but they're little-endian so the resulting bit order (not byte, mind you; you're happily using the same word for both bits and bytes). But what the calculation should be based on? What do you want to achieve?

You showed only one example which is a power of 2 so it gives you just one set bit in your whole 16-bit sequence. But what if you had 0x63 0x3A?

0x63 is 01100011, 0x3A is 00111010

As this is little-endian, the resulting bit-stream would be

00111010 01100011

And what now?

You want the position of first non-zero bit from the right?

And what does it have to do with "lookup"?

[smanojkumar]

Hello @PickleRick ,

   Sorry for the mistake!

   I have edited in the post,

Considering 0*63 is 01100011

Seperating the byte as with 2positions

byte 0 - 01 -> 0000 0001
byte 1 - 10 -> 0001 0000
byte 2 - 00 -> 0000 0000
byte 3 - 11 -> 0001 0001

Combining it byte 3 byte 2 byte 1 byte 0 - 0001 0001 0000 0000 0001 0000 0000 0001

Counting the non zero postions  - 0, 12, 24, 28

These were the answers.

please let me know if there are anything.

Thanks in advance!

[PickleRick]

OK. Fully honestly, I don't see the point in doing so. You're adding some synthetic zeros and then count something which will easily be divisible by 4 (because when you're preparing your bitstring he ones must be separated by multiplies of 4).

You're doing some strange bit swapping (and it's completely inconsistent - your original example had something that resembled a 16-bits little-endian integer which you "converted" to 32-bit value and from my single bit 0x63 value you also built a 32-bit value but obviously in a different way).

So, to be absolutely frank, it doesn't make much sense.

A completely separate thing is why would you want to do it in splunk? (actually the easiest solution could be to implement an external lookup using a python script).