Splunk Search

Conver to Hexadecimal code to bit value

smanojkumar
Contributor

Hello,
   I would like to convert my hexadecimal code to a bit value based on this calculation. 

Hex code - 0002

Seperate 2 bytes each  00/02

2 Byte bitmask
Byte 0: HEX = 00 - 0000 0000
Byte 1: HEX = 02 - 0000 0010

Byte 1 Byte0 - 0000 0010 0000 0000

calculate the non zero th position values from right side

Byte combination  - 0000 0010 0000 0000
Position -                      9 8765  4321

At position 10, we got 1 while counting from right side. so the bit value is 9.


I need to calculate this in splunk, where the HEX_Code is the value from the lookup.

Thanks in Advance!
Happy Splunking!



Labels (1)
0 Karma
1 Solution

yuanliu
SplunkTrust
SplunkTrust

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by 

@PickleRick Lol you wouldn't believe how much time I spent trying to decipher the OP's intent from the various posted replies to everybody's attempt to help.  After hours of scrolling up and down, back and forth, I distilled the instructions into the following algorithm given an even numbered HEX string, e.g., aabbcc

  1. Break the string into 2-HEX chunks. (OP used the term 2-bytes - I realize that is actually 4-bytes)
  2. Convert each chunk into binary.
  3. Reverse the order of the binary chunks.
  4. Count the positions of nonzero bits of the full reversed binary string from the right.

(As I said, I can't think of a practical purpose of this exercise. By the way, to anyone who is going to ask a question here, even though I strongly encourage describing problem without SPL first, please make the description as algorithmic as possible.)  As a weird game, this applies to any even-length HEX string.  Here's a sequence of up to 16 HEX characters.

hexpadded_binarynonzero_bits
01000000010
0002
00000010
00000000
9
000003
00000011
00000000
00000000
16
17
00000004
00000100
00000000
00000000
00000000
26
0000000005
00000101
00000000
00000000
00000000
00000000
32
34
000000000006
00000110
00000000
00000000
00000000
00000000
00000000
41
42
00000000000007
00000111
00000000
00000000
00000000
00000000
00000000
00000000
48
49
50
0000000000000008
00001000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
59
90000000000000
00000000
00000000
00000000
00000000
00000000
00000000
10010000
4
7
a00000000000
00000000
00000000
00000000
00000000
00000000
10100000
5
7
b000000000
00000000
00000000
00000000
00000000
10110000
4
5
7
c0000000
00000000
00000000
00000000
11000000
6
7
d00000
00000000
00000000
11010000
4
6
7
e000
00000000
11100000
5
6
7
f011110000
4
5
6
7

Another thing I realize is that I must handle 2-HEX (single-chunk) specially.  Here is the emulation code

 

| makeresults format=csv data="hex
01
0002
000003
00000004
0000000005
000000000006
00000000000007
0000000000000008
90000000000000
a00000000000
b000000000
c0000000
d00000
e000
f0"
``` data emulation above ```
| eval idx = mvrange(0, len(hex) / 2)
| eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2)))
| eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary")))
| eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary)))
| eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1
| foreach reverse_bits mode=multivalue
    [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))]
| fields hex padded_binary nonzero_bits

 

(Technically this works for odd number of HEX characters, too, if OP can define where to split.) 

View solution in original post

tscroggins
Influencer

Hi @smanojkumar,

This is a 16-bit adaptation of Gaudet's algorithm from Hacker's Delight Second Edition (Warren, 2013):

| makeresults
| eval HEX_Code="0002"
``` convert to number ```
| eval x=tonumber(HEX_Code, 16)
``` swap bytes ```
| eval x=bit_shift_right(x, 8)+bit_and(bit_shift_left(x, 8), 65280)
``` calculate number of trailing zeros (ntz) ```
| eval y=bit_and(x, 65535-x+1)
| eval bz=if(y>0, 0, 1), b3=if(bit_and(y, 255)>0, 0, 8), b2=if(bit_and(y, 3855)>0, 0, 4), b1=if(bit_and(y, 13107)>0, 0, 2), b0=if(bit_and(y, 21845)>0, 0, 1)
| eval ntz=bz+b3+b2+b1+b0
``` ntz=9 ```

 

0 Karma

smanojkumar
Contributor

Hello @tscroggins ,
   Thanks for your reply!

   I'm having this error - "Error in 'EvalCommand': The 'bit_shift_right' function is unsupported or undefined."

Can you help in resolving this error.

Thanks in Advance!

0 Karma

tscroggins
Influencer

Here's an alternative that uses a few helper macros to replace the bitwise eval functions. Bit rotate functions would be a nice addition to Splunk, as would a parameter on all bitwise functions to specify width.

| makeresults
| eval HEX_Code="0002"
``` convert to number ```
| eval x=tonumber(HEX_Code, 16)
``` swap bytes ```
| eval t=`bitshl(x, 8)`, x=`bitshr(x, 8)`+`bitand_16(t, 65280)`
``` calculate number of trailing zeros (ntz) ```
| eval t=65535-x+1, y=`bitand_16(x, t)`
| eval bz=if(y>0, 0, 1), b3=if(`bitand_16(y, 255)`>0, 0, 8), b2=if(`bitand_16(y, 3855)`>0, 0, 4), b1=if(`bitand_16(y, 13107)`>0, 0, 2), b0=if(`bitand_16(y, 21845)`>0, 0, 1)
| eval ntz=bz+b3+b2+b1+b0
``` ntz=9 ```
# macros.conf

[bitand_16(2)]
args = x, y
definition = sum(1 * (floor($x$ / 1) % 2) * (floor($y$ / 1) % 2), 2 * (floor($x$ / 2) % 2) * (floor($y$ / 2) % 2), 4 * (floor($x$ / 4) % 2) * (floor($y$ / 4) % 2), 8 * (floor($x$ / 😎 % 2) * (floor($y$ / 😎 % 2), 16 * (floor($x$ / 16) % 2) * (floor($y$ / 16) % 2), 32 * (floor($x$ / 32) % 2) * (floor($y$ / 32) % 2), 64 * (floor($x$ / 64) % 2) * (floor($y$ / 64) % 2), 128 * (floor($x$ / 128) % 2) * (floor($y$ / 128) % 2), 256 * (floor($x$ / 256) % 2) * (floor($y$ / 256) % 2), 512 * (floor($x$ / 512) % 2) * (floor($y$ / 512) % 2), 1024 * (floor($x$ / 1024) % 2) * (floor($y$ / 1024) % 2), 2048 * (floor($x$ / 2048) % 2) * (floor($y$ / 2048) % 2), 4096 * (floor($x$ / 4096) % 2) * (floor($y$ / 4096) % 2), 8192 * (floor($x$ / 8192) % 2) * (floor($y$ / 8192) % 2), 16384 * (floor($x$ / 16384) % 2) * (floor($y$ / 16384) % 2), 32768 * (floor($x$ / 32768) % 2) * (floor($y$ / 32768) % 2))
iseval = 0

[bitshl(2)]
args = x, k
definition = floor(pow(2, $k$) * $x$)
iseval = 0

[bitshr(2)]
args = x, k
definition = floor(pow(2, -$k$) * $x$)
iseval = 0

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

bit-wise functions only came into Splunk Enterprise in 9.2.0 and in 9.1 in Cloud Services (according to the documentation) - which version of Splunk are you using?

0 Karma

yuanliu
SplunkTrust
SplunkTrust

@smanojkumar Can you confirm that results you are looking for are like the following?

hex
padded_binary
nonzero_bits
0002
00000010
00000000
9
00200100
00000000
00000001
00100000
00000000
13
16
01100011
00010001
00000000
00010000
00000001
0
12
24
28

This sounds like some data compression game.  I can't think of a practical reason to do this in SPL.  Is this some sort of homework?

Anyway, here is a more or less literal way to interpret your instructions:

 

 

| eval idx = mvrange(0, len(hex) / 2)
| eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2)))
| eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary")))
| eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary)))
| eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1
| foreach reverse_bits mode=multivalue
    [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))]
| fields hex padded_binary nonzero_bits

 

 

Note mvreverse on padded binary is sort of expensive and can be avoided by arithmetics if there are lots of data.

Here is the emulation of the three examples you give:

 

 

| makeresults format=csv data="hex
0002
00200100
01100011"
``` data emulation above ```

 

 

Apply the algorithm to this emulation gives the results tabulated at the top.

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by @smanojkumar 's explanation as to how the algorithm is supposed to work. Does it work on 16-bit integers only? Does it work on any length stream of data? Does it always produce 32-bit integers? Or does the result grow with the length of the argument? It's so badly specified...

0 Karma

yuanliu
SplunkTrust
SplunkTrust

@yuanliu 01100011 was _not_ hex. It was binary for 0x63. That's why I'm completely confused by 

@PickleRick Lol you wouldn't believe how much time I spent trying to decipher the OP's intent from the various posted replies to everybody's attempt to help.  After hours of scrolling up and down, back and forth, I distilled the instructions into the following algorithm given an even numbered HEX string, e.g., aabbcc

  1. Break the string into 2-HEX chunks. (OP used the term 2-bytes - I realize that is actually 4-bytes)
  2. Convert each chunk into binary.
  3. Reverse the order of the binary chunks.
  4. Count the positions of nonzero bits of the full reversed binary string from the right.

(As I said, I can't think of a practical purpose of this exercise. By the way, to anyone who is going to ask a question here, even though I strongly encourage describing problem without SPL first, please make the description as algorithmic as possible.)  As a weird game, this applies to any even-length HEX string.  Here's a sequence of up to 16 HEX characters.

hexpadded_binarynonzero_bits
01000000010
0002
00000010
00000000
9
000003
00000011
00000000
00000000
16
17
00000004
00000100
00000000
00000000
00000000
26
0000000005
00000101
00000000
00000000
00000000
00000000
32
34
000000000006
00000110
00000000
00000000
00000000
00000000
00000000
41
42
00000000000007
00000111
00000000
00000000
00000000
00000000
00000000
00000000
48
49
50
0000000000000008
00001000
00000000
00000000
00000000
00000000
00000000
00000000
00000000
59
90000000000000
00000000
00000000
00000000
00000000
00000000
00000000
10010000
4
7
a00000000000
00000000
00000000
00000000
00000000
00000000
10100000
5
7
b000000000
00000000
00000000
00000000
00000000
10110000
4
5
7
c0000000
00000000
00000000
00000000
11000000
6
7
d00000
00000000
00000000
11010000
4
6
7
e000
00000000
11100000
5
6
7
f011110000
4
5
6
7

Another thing I realize is that I must handle 2-HEX (single-chunk) specially.  Here is the emulation code

 

| makeresults format=csv data="hex
01
0002
000003
00000004
0000000005
000000000006
00000000000007
0000000000000008
90000000000000
a00000000000
b000000000
c0000000
d00000
e000
f0"
``` data emulation above ```
| eval idx = mvrange(0, len(hex) / 2)
| eval reverse2hex = mvreverse(mvmap(idx, substr(hex, idx*2 + 1, 2)))
| eval ASbinary=if(idx < 1, tostring(tonumber(reverse2hex,16),"binary"), mvmap(reverse2hex, tostring(tonumber(reverse2hex,16),"binary")))
| eval padded_binary = if(idx < 1, printf("%08d", ASbinary), mvmap(ASbinary, printf("%08d", ASbinary)))
| eval reverse_bits = mvreverse(mvmap(padded_binary, split(padded_binary, ""))), position = -1
| foreach reverse_bits mode=multivalue
    [eval position = position + 1, nonzero_bits = if(<<ITEM>> == 0, nonzero_bits, mvappend(nonzero_bits, position))]
| fields hex padded_binary nonzero_bits

 

(Technically this works for odd number of HEX characters, too, if OP can define where to split.) 

smanojkumar
Contributor

Hi @yuanliu & @ITWhisperer  & @tscroggins  & @PickleRick  & @dural_yyz ,

    Thanks everyone for your time, it works for me.


Thanks in Advance!

0 Karma

smanojkumar
Contributor

Hello @yuanliu ,
   Thanks for your response!

    I'm having this error "Error in 'EvalCommand': The arguments to the 'tostring' function are invalid.", can you please help me in this.


Thanks in advance!

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

"binary" only came into Splunk Enterprise in 9.2.0 and doesn't appear to be in Cloud Services yet (according to the documentation)

0 Karma

smanojkumar
Contributor

Hello @ITWhisperer ,
   Thanks for that.
   I'm currebtly using Splunk Enterprise with Version 9.1.1, So thats the reason.

   Any alternative way to work on this with this version?

Thanks for pointing it out.

regards,
Manoj Kumar S

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Having converted the number to hex, perform 16 replacements, starting with 0, then 1, replacing the hex digit with the corresponding binary equivalent.

0 Karma

smanojkumar
Contributor

Hello @ITWhisperer ,

   Thanks for your resposne!

   If you don't mind changing the code as well.

Thansk a lot for your resposne!

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Here are the first four, I am sure you can workout from this how to do the other 12 hex digits

| eval ASbinary=if(idx < 1, replace(replace(replace(replace(reverse2hex,"0","0000"),"1","0001"),"2","0010"),"3","0011"), mvmap(reverse2hex, replace(replace(replace(replace(reverse2hex,"0","0000"),"1","0001"),"2","0010"),"3","0011")))
0 Karma

smanojkumar
Contributor

Hi @ITWhisperer ,

   Thanks, It works on that place.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Nice job. I'm stil not convinced it's how it was supposed to work 😉

0 Karma

smanojkumar
Contributor

Another example

Hex code - 00200100

Seperate 2 bytes each  00/20/01/00

4 Byte bitmask
Byte 0: HEX = 00 -  0000 0000
Byte 1: HEX = 20 - 0010 0000
Byte 2: HEX = 01 - 0000 0001
Byte 3: HEX = 00 - 0000 0000


Byte 3 Byte 2 Byte1 Byte 0 - 0000 0000 0000 0001 0010 0000 0000 0000

calculate the non zero th position values from right side

Byte combination  - 0000 0000 0000 0001 0010 0000 0000 0000
Position -                                                               16       13        ...9 8765  4321

At position 14 and 17, we got 1 while counting from right side. so the bit value is 13 and 16.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. This is confusing. You have four hexadecimal digits but they're little-endian so the resulting bit order (not byte, mind you; you're happily using the same word for both bits and bytes). But what the calculation should be based on? What do you want to achieve?

You showed only one example which is a power of 2 so it gives you just one set bit in your whole 16-bit sequence. But what if you had 0x63 0x3A?

0x63 is 01100011, 0x3A is 00111010

As this is little-endian, the resulting bit-stream would be

00111010 01100011

And what now?

You want the position of first non-zero bit from the right?

And what does it have to do with "lookup"?

0 Karma

smanojkumar
Contributor

Hello @PickleRick ,

   Sorry for the mistake!

   I have edited in the post,

Considering 0*63 is 01100011

Seperating the byte as with 2positions

byte 0 - 01 -> 0000 0001
byte 1 - 10 -> 0001 0000
byte 2 - 00 -> 0000 0000
byte 3 - 11 -> 0001 0001

Combining it byte 3 byte 2 byte 1 byte 0 - 0001 0001 0000 0000 0001 0000 0000 0001

Counting the non zero postions  - 0, 12, 24, 28

These were the answers.

please let me know if there are anything.

Thanks in advance!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. Fully honestly, I don't see the point in doing so. You're adding some synthetic zeros and then count something which will easily be divisible by 4 (because when you're preparing your bitstring he ones must be separated by multiplies of 4).

You're doing some strange bit swapping (and it's completely inconsistent - your original example had something that resembled a 16-bits little-endian integer which you "converted" to 32-bit value and from my single bit 0x63 value you also built a 32-bit value but obviously in a different way).

So, to be absolutely frank, it doesn't make much sense.

A completely separate thing is why would you want to do it in splunk? (actually the easiest solution could be to implement an external lookup using a python script).

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...