topic Conver to Hexadecimal code to bit value in Splunk Search

Conver to Hexadecimal code to bit value

smanojkumar — Fri, 20 Sep 2024 18:35:41 GMT

Hello,
I would like to convert my hexadecimal code to a bit value based on this calculation.

Hex code - 0002

Seperate 2 bytes each 00/02

2 Byte bitmask
Byte 0: HEX = 00 - 0000 0000
Byte 1: HEX = 02 - 0000 0010

Byte 1 Byte0 - 0000 0010 0000 0000

calculate the non zero th position values from right side

Byte combination - 0000 0010 0000 0000
Position - 9 8765 4321

At position 10, we got 1 while counting from right side. so the bit value is 9.

I need to calculate this in splunk, where the HEX_Code is the value from the lookup.

Thanks in Advance!
Happy Splunking!

Re: Conver to Hexadecimal code to bit value

ITWhisperer — Fri, 20 Sep 2024 11:50:55 GMT

It is not clear exactly what you are trying to do - is the hex code to be treated as if it is least significant byte first (left-most) and most significant byte second, but the bits in the byte are to be treated as most significant bit first (left-mode). Is that right?

Do you just want to know the bit position of the least significant bit?

Re: Conver to Hexadecimal code to bit value

dural_yyz — Fri, 20 Sep 2024 13:55:17 GMT

| makeresults format=csv data="bit1,bit2 0000,0002 000f,0088 00af,00de 00bd,003c" | fields bit1 bit2 | eval bit1ASnumber=tonumber(bit1,16), bit2ASnumber=tonumber(bit2,16) | eval bit1ASbinary=tostring(bit1ASnumber,"binary"), bit2ASbinary=tostring(bit2ASnumber,"binary") | table bit1 bit2 bit*ASnum* bit*ASbin*

bit1	bit2	bit1ASnumber	bit2ASnumber	bit1ASbinary	bit2ASbinary
0000	0002	0	2	0	10
000f	0088	15	136	1111	10001000
00af	00de	175	222	10101111	11011110
00bd	003c	189	60	10111101	111100

Ok I can get you as far as converting to binary but the results of the binary do not include leading 0's to always make an 8 character string/number. Since your example had 4 characters for the hex code those values are treated as string so first convert to a number before converting to binary as attempting to go straight will fail when your source as a mix of alphanumeric characters.

Obviously without the leading zeros when you concatenate the two values as strings you will lose some positions you need to count. Also I didn't bother with sorting out the count how many zero's right of the last occurring 1 but essentially that's what comes after inserting your leading zero's

Re: Conver to Hexadecimal code to bit value

PickleRick — Fri, 20 Sep 2024 14:54:00 GMT

OK. This is confusing. You have four hexadecimal digits but they're little-endian so the resulting bit order (not byte, mind you; you're happily using the same word for both bits and bytes). But what the calculation should be based on? What do you want to achieve?

You showed only one example which is a power of 2 so it gives you just one set bit in your whole 16-bit sequence. But what if you had 0x63 0x3A?

0x63 is 01100011, 0x3A is 00111010

As this is little-endian, the resulting bit-stream would be

00111010 01100011

And what now?

You want the position of first non-zero bit from the right?

And what does it have to do with "lookup"?

Re: Conver to Hexadecimal code to bit value

smanojkumar — Fri, 20 Sep 2024 18:35:16 GMT

Another example

Hex code - 00200100

Seperate 2 bytes each 00/20/01/00

4 Byte bitmask
Byte 0: HEX = 00 - 0000 0000
Byte 1: HEX = 20 - 0010 0000
Byte 2: HEX = 01 - 0000 0001
Byte 3: HEX = 00 - 0000 0000

Byte 3 Byte 2 Byte1 Byte 0 - 0000 0000 0000 0001 0010 0000 0000 0000

calculate the non zero th position values from right side

Byte combination - 0000 0000 0000 0001 0010 0000 0000 0000
Position - 16 13 ...9 8765 4321

At position 14 and 17, we got 1 while counting from right side. so the bit value is 13 and 16.

Re: Conver to Hexadecimal code to bit value

smanojkumar — Fri, 20 Sep 2024 18:39:26 GMT

Hi @ITWhisperer ,

Thanks for your time!

Here is an another example

Re: Conver to Hexadecimal code to bit value

smanojkumar — Fri, 20 Sep 2024 18:39:50 GMT

Hi @dural_yyz ,

Thanks for your reply!

please look at this example as well, also we are having hex_code as a single field value, from that field we are seperating bit 1 and bit 2,

bit position of the least significant bit

Another example

Hex code - 00200100

Seperate 2 bytes each 00/20/01/00

4 Byte bitmask
Byte 0: HEX = 00 - 0000 0000
Byte 1: HEX = 20 - 0010 0000
Byte 2: HEX = 01 - 0000 0001
Byte 3: HEX = 00 - 0000 0000

Byte 3 Byte 2 Byte1 Byte 0 - 0000 0000 0000 0001 0010 0000 0000 0000

calculate the non zero th position values from right side

Byte combination - 0000 0000 0000 0001 0010 0000 0000 0000
Position - 16 13 ...9 8765 4321

At position 14 and 17, we got 1 while counting from right side. so the bit value is 13 and 16.

thanks!

Re: Conver to Hexadecimal code to bit value

smanojkumar — Fri, 20 Sep 2024 18:46:48 GMT

Hello @PickleRick ,

Sorry for the mistake!

I have edited in the post,

Considering 0*63 is 01100011

Seperating the byte as with 2positions

byte 0 - 01 -> 0000 0001
byte 1 - 10 -> 0001 0000
byte 2 - 00 -> 0000 0000
byte 3 - 11 -> 0001 0001

Combining it byte 3 byte 2 byte 1 byte 0 - 0001 0001 0000 0000 0001 0000 0000 0001

Counting the non zero postions - 0, 12, 24, 28

These were the answers.

please let me know if there are anything.

Thanks in advance!

Re: Conver to Hexadecimal code to bit value

PickleRick — Fri, 20 Sep 2024 20:18:05 GMT

OK. Fully honestly, I don't see the point in doing so. You're adding some synthetic zeros and then count something which will easily be divisible by 4 (because when you're preparing your bitstring he ones must be separated by multiplies of 4).

You're doing some strange bit swapping (and it's completely inconsistent - your original example had something that resembled a 16-bits little-endian integer which you "converted" to 32-bit value and from my single bit 0x63 value you also built a 32-bit value but obviously in a different way).

So, to be absolutely frank, it doesn't make much sense.

A completely separate thing is why would you want to do it in splunk? (actually the easiest solution could be to implement an external lookup using a python script).

Re: Conver to Hexadecimal code to bit value

yuanliu — Sun, 22 Sep 2024 01:23:08 GMT

@smanojkumar Can you confirm that results you are looking for are like the following?

hex	padded_binary	nonzero_bits
0002	00000010 00000000	9
00200100	00000000 00000001 00100000 00000000	13 16
01100011	00010001 00000000 00010000 00000001	0 12 24 28

This sounds like some data compression game. I can't think of a practical reason to do this in SPL. Is this some sort of homework?

Anyway, here is a more or less literal way to interpret your instructions:

| eval idx = mvrange(0, len(hex) / 2) | eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2))) | eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary"))) | eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary))) | eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1 | foreach reverse_bits mode=multivalue [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))] | fields hex padded_binary nonzero_bits

Note mvreverse on padded binary is sort of expensive and can be avoided by arithmetics if there are lots of data.

Here is the emulation of the three examples you give:

| makeresults format=csv data="hex 0002 00200100 01100011" ``` data emulation above ```

Apply the algorithm to this emulation gives the results tabulated at the top.

Re: Conver to Hexadecimal code to bit value

PickleRick — Sat, 21 Sep 2024 11:02:33 GMT

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by @smanojkumar 's explanation as to how the algorithm is supposed to work. Does it work on 16-bit integers only? Does it work on any length stream of data? Does it always produce 32-bit integers? Or does the result grow with the length of the argument? It's so badly specified...

Re: Conver to Hexadecimal code to bit value

yuanliu — Sun, 22 Sep 2024 01:20:10 GMT

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by

@PickleRick Lol you wouldn't believe how much time I spent trying to decipher the OP's intent from the various posted replies to everybody's attempt to help. After hours of scrolling up and down, back and forth, I distilled the instructions into the following algorithm given an even numbered HEX string, e.g., aabbcc

Break the string into 2-HEX chunks. (OP used the term 2-bytes - I realize that is actually 4-bytes)
Convert each chunk into binary.
Reverse the order of the binary chunks.
Count the positions of nonzero bits of the full reversed binary string from the right.

(As I said, I can't think of a practical purpose of this exercise. By the way, to anyone who is going to ask a question here, even though I strongly encourage describing problem without SPL first, please make the description as algorithmic as possible.) As a weird game, this applies to any even-length HEX string. Here's a sequence of up to 16 HEX characters.

hex	padded_binary	nonzero_bits
01	00000001	0
0002	00000010 00000000	9
000003	00000011 00000000 00000000	16 17
00000004	00000100 00000000 00000000 00000000	26
0000000005	00000101 00000000 00000000 00000000 00000000	32 34
000000000006	00000110 00000000 00000000 00000000 00000000 00000000	41 42
00000000000007	00000111 00000000 00000000 00000000 00000000 00000000 00000000	48 49 50
0000000000000008	00001000 00000000 00000000 00000000 00000000 00000000 00000000 00000000	59
90000000000000	00000000 00000000 00000000 00000000 00000000 00000000 10010000	4 7
a00000000000	00000000 00000000 00000000 00000000 00000000 10100000	5 7
b000000000	00000000 00000000 00000000 00000000 10110000	4 5 7
c0000000	00000000 00000000 00000000 11000000	6 7
d00000	00000000 00000000 11010000	4 6 7
e000	00000000 11100000	5 6 7
f0	11110000	4 5 6 7

Another thing I realize is that I must handle 2-HEX (single-chunk) specially. Here is the emulation code

| makeresults format=csv data="hex 01 0002 000003 00000004 0000000005 000000000006 00000000000007 0000000000000008 90000000000000 a00000000000 b000000000 c0000000 d00000 e000 f0" ``` data emulation above ``` | eval idx = mvrange(0, len(hex) / 2) | eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2))) | eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary"))) | eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary))) | eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1 | foreach reverse_bits mode=multivalue [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))] | fields hex padded_binary nonzero_bits

(Technically this works for odd number of HEX characters, too, if OP can define where to split.)

Re: Conver to Hexadecimal code to bit value

PickleRick — Sun, 22 Sep 2024 16:16:10 GMT

Nice job. I'm stil not convinced it's how it was supposed to work 😉

Re: Conver to Hexadecimal code to bit value

tscroggins — Mon, 23 Sep 2024 00:37:19 GMT

Hi @smanojkumar,

This is a 16-bit adaptation of Gaudet's algorithm from Hacker's Delight Second Edition (Warren, 2013):

| makeresults | eval HEX_Code="0002" ``` convert to number ``` | eval x=tonumber(HEX_Code, 16) ``` swap bytes ``` | eval x=bit_shift_right(x, 8)+bit_and(bit_shift_left(x, 8), 65280) ``` calculate number of trailing zeros (ntz) ``` | eval y=bit_and(x, 65535-x+1) | eval bz=if(y>0, 0, 1), b3=if(bit_and(y, 255)>0, 0, 8), b2=if(bit_and(y, 3855)>0, 0, 4), b1=if(bit_and(y, 13107)>0, 0, 2), b0=if(bit_and(y, 21845)>0, 0, 1) | eval ntz=bz+b3+b2+b1+b0 ``` ntz=9 ```

Re: Conver to Hexadecimal code to bit value

smanojkumar — Mon, 23 Sep 2024 06:09:48 GMT

Hello @tscroggins ,
Thanks for your reply!

I'm having this error - "Error in 'EvalCommand': The 'bit_shift_right' function is unsupported or undefined."

Can you help in resolving this error.

Thanks in Advance!

Re: Conver to Hexadecimal code to bit value

smanojkumar — Mon, 23 Sep 2024 07:07:27 GMT

Hello @yuanliu ,
Thanks for your response!

I'm having this error "Error in 'EvalCommand': The arguments to the 'tostring' function are invalid.", can you please help me in this.

Thanks in advance!

Re: Conver to Hexadecimal code to bit value

ITWhisperer — Mon, 23 Sep 2024 07:28:47 GMT

"binary" only came into Splunk Enterprise in 9.2.0 and doesn't appear to be in Cloud Services yet (according to the documentation)

Re: Conver to Hexadecimal code to bit value

ITWhisperer — Mon, 23 Sep 2024 07:33:37 GMT

bit-wise functions only came into Splunk Enterprise in 9.2.0 and in 9.1 in Cloud Services (according to the documentation) - which version of Splunk are you using?

Re: Conver to Hexadecimal code to bit value

smanojkumar — Mon, 23 Sep 2024 08:39:55 GMT

Hello @ITWhisperer ,
Thanks for that.
I'm currebtly using Splunk Enterprise with Version 9.1.1, So thats the reason.

Any alternative way to work on this with this version?

Thanks for pointing it out.

regards,
Manoj Kumar S

Re: Conver to Hexadecimal code to bit value

ITWhisperer — Mon, 23 Sep 2024 09:01:17 GMT

Having converted the number to hex, perform 16 replacements, starting with 0, then 1, replacing the hex digit with the corresponding binary equivalent.