How do I extract this field with regex for Nmap result?

[bnybln030]

Hi i want to extract the mac_algorithms field with regex from a nmap scan result. Does anyone have an idea how it works best? I've tried a few things, not all fields are found in Splunk. 

Here you can see my example: https://regex101.com/r/eJ16fA/1

Here my nmap-scanning example:

kex_algorithms: (8)
[email protected]
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group-exchange-sha256
diffie-hellman-group-exchange-sha1
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
server_host_key_algorithms: (4)
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ssh-ed25519
encryption_algorithms: (14)
aes128-ctr
aes192-ctr
aes256-ctr
arcfour256
arcfour128
[email protected]
aes128-cbc
3des-cbc
blowfish-cbc
cast128-cbc
aes192-cbc
aes256-cbc
arcfour
[email protected]
mac_algorithms: (19)
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
hmac-md5
hmac-sha1
[email protected]
[email protected]
hmac-sha2-256
hmac-sha2-512
hmac-ripemd160
[email protected]
hmac-sha1-96
hmac-md5-96
compression_algorithms: (2)
none
[email protected]"

[ITWhisperer]

I am not sure what the issue is here - your regex101 example seems to show all 19 algorithms have been matched. What is your expected/required result?

[bnybln030]

The problem is that not all values ​​are found in splunk. I only see 8 but in the example on regex101 there are 19 different ones. 

Here you can see my result:

test.png

[yuanliu]

• Have you tested that your events contain all 19 MAC algorithms? (This can be tested by searching for those that you believe are missed by Splunk search.)
• Is it possible that MAC may appear without leading space (hence not matching "^\s+") or appear after non-whitespace characters?  MAC algorithm patterns are unique enough that you can simply use "\b" to delimit it.  This example illustrates that Splunk does extract all 19 algos if events contain them all.

| makeresults
| eval event=split("  kex_algorithms: (8)
      [email protected]
      ecdh-sha2-nistp256
      ecdh-sha2-nistp384
      ecdh-sha2-nistp521
      diffie-hellman-group-exchange-sha256
      diffie-hellman-group-exchange-sha1
      diffie-hellman-group14-sha1
      diffie-hellman-group1-sha1
  server_host_key_algorithms: (4)
      ssh-rsa
      ssh-dss
      ecdsa-sha2-nistp256
      ssh-ed25519
  encryption_algorithms: (14)
      aes128-ctr
      aes192-ctr
      aes256-ctr
      arcfour256
      arcfour128
      [email protected]
      aes128-cbc
      3des-cbc
      blowfish-cbc
      cast128-cbc
      aes192-cbc
      aes256-cbc
      arcfour
      [email protected]
  mac_algorithms: (19)
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      hmac-md5
      hmac-sha1
      [email protected]
      [email protected]
      hmac-sha2-256
      hmac-sha2-512
      hmac-ripemd160
      [email protected]
      hmac-sha1-96
      hmac-md5-96
  compression_algorithms: (2)
      none
      [email protected]", "
")
| mvexpand event
``` event simulation above ```

| rex field=event max_match=0 "\b(?<test>[uh]mac-[\w\-\.@]+)"​

Output from the simulation is

event	test
kex_algorithms: (8)
[email protected]
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group-exchange-sha256
diffie-hellman-group-exchange-sha1
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
server_host_key_algorithms: (4)
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ssh-ed25519
encryption_algorithms: (14)
aes128-ctr
aes192-ctr
aes256-ctr
arcfour256
arcfour128
[email protected]
aes128-cbc
3des-cbc
blowfish-cbc
cast128-cbc
aes192-cbc
aes256-cbc
arcfour
[email protected]
mac_algorithms: (19)
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
[email protected]	[email protected]
hmac-md5	hmac-md5
hmac-sha1	hmac-sha1
[email protected]	[email protected]
[email protected]	[email protected]
hmac-sha2-256	hmac-sha2-256
hmac-sha2-512	hmac-sha2-512
hmac-ripemd160	hmac-ripemd160
[email protected]	[email protected]
hmac-sha1-96	hmac-sha1-96
hmac-md5-96	hmac-md5-96
compression_algorithms: (2)
none
[email protected]

Note

1. "^\s+\K" is really just "^\s+".
2. An explicit "-" after "(hmac|umac)" better expresses the MAC format, thus reduces potential false positives.

[bnybln030]

I see in the field extractor that only the first entry is recognized....

test4.png

[yuanliu]

---

@bnybln030 wrote:

I see in the field extractor that only the first entry is recognized....

---

Can you post the complete filter line in your search? (From "|" all the way to  the end or next "|")  The screenshot only captures "\b(?<test>[hu]mac-[\w\-\.@]+)", which doesn't contain a valid SPL command.  My posted filter is

| rex field=event max_match=0 "\b(?<test>[hu]mac-[\w\-\.@]+)"

The argument "max_match=0" makes sure to match all entries in a multi-line event.  Side note: If your input  is as that screenshot shows, containing multiple lines of MAC algorithms, literally copying that regex101 into rex command without max_match=0 will only extract the first occurrence.  This could explain why you only got 8 in the first place. (See "Optional arguments" in rex.

Here is a test with all 19 algos in one event:

• With max_match=0

| makeresults
| eval event="  kex_algorithms: (8)
      [email protected]
      ecdh-sha2-nistp256
      ecdh-sha2-nistp384
      ecdh-sha2-nistp521
      diffie-hellman-group-exchange-sha256
      diffie-hellman-group-exchange-sha1
      diffie-hellman-group14-sha1
      diffie-hellman-group1-sha1
  server_host_key_algorithms: (4)
      ssh-rsa
      ssh-dss
      ecdsa-sha2-nistp256
      ssh-ed25519
  encryption_algorithms: (14)
      aes128-ctr
      aes192-ctr
      aes256-ctr
      arcfour256
      arcfour128
      [email protected]
      aes128-cbc
      3des-cbc
      blowfish-cbc
      cast128-cbc
      aes192-cbc
      aes256-cbc
      arcfour
      [email protected]
  mac_algorithms: (19)
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      hmac-md5
      hmac-sha1
      [email protected]
      [email protected]
      hmac-sha2-256
      hmac-sha2-512
      hmac-ripemd160
      [email protected]
      hmac-sha1-96
      hmac-md5-96
  compression_algorithms: (2)
      none
      [email protected]"
``` event simulation above ```

| rex field=event max_match=0 "\b(?<test>[hu]mac-[\w\-\.@]+)"
| eval algocount=mvcount(test)​
| table algocount event test

algocount	event	test
19	kex_algorithms: (8) [email protected] ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1 diffie-hellman-group1-sha1 server_host_key_algorithms: (4) ssh-rsa ssh-dss ecdsa-sha2-nistp256 ssh-ed25519 encryption_algorithms: (14) aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 [email protected] aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour [email protected] mac_algorithms: (19) [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] hmac-md5 hmac-sha1 [email protected] [email protected] hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 [email protected] hmac-sha1-96 hmac-md5-96 compression_algorithms: (2) none [email protected]	[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] hmac-md5 hmac-sha1 [email protected] [email protected] hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 [email protected] hmac-sha1-96 hmac-md5-96

• No max_match=0

| makeresults
| eval event="  kex_algorithms: (8)
      [email protected]
      ecdh-sha2-nistp256
      ecdh-sha2-nistp384
      ecdh-sha2-nistp521
      diffie-hellman-group-exchange-sha256
      diffie-hellman-group-exchange-sha1
      diffie-hellman-group14-sha1
      diffie-hellman-group1-sha1
  server_host_key_algorithms: (4)
      ssh-rsa
      ssh-dss
      ecdsa-sha2-nistp256
      ssh-ed25519
  encryption_algorithms: (14)
      aes128-ctr
      aes192-ctr
      aes256-ctr
      arcfour256
      arcfour128
      [email protected]
      aes128-cbc
      3des-cbc
      blowfish-cbc
      cast128-cbc
      aes192-cbc
      aes256-cbc
      arcfour
      [email protected]
  mac_algorithms: (19)
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      [email protected]
      hmac-md5
      hmac-sha1
      [email protected]
      [email protected]
      hmac-sha2-256
      hmac-sha2-512
      hmac-ripemd160
      [email protected]
      hmac-sha1-96
      hmac-md5-96
  compression_algorithms: (2)
      none
      [email protected]"
``` event simulation above ```

| rex field=event "\b(?<test>[hu]mac-[\w\-\.@]+)"
| eval algocount=mvcount(test)
| table algocount event test​

algocount	event	test
19	kex_algorithms: (8) [email protected] ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1 diffie-hellman-group1-sha1 server_host_key_algorithms: (4) ssh-rsa ssh-dss ecdsa-sha2-nistp256 ssh-ed25519 encryption_algorithms: (14) aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 [email protected] aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour [email protected] mac_algorithms: (19) [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] hmac-md5 hmac-sha1 [email protected] [email protected] hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 [email protected] hmac-sha1-96 hmac-md5-96 compression_algorithms: (2) none [email protected]	[email protected]

[bnybln030]

You are right, on my screenshot you see the complete search. With "max_match=0" i get the right results:

test5.png

If I dont set "max_match=0" then I get exactly the same result as with field extractor:

test6.png

[ITWhisperer]

You probably need to enable multivalue field extraction?

[bnybln030]

How can i do it?

[bnybln030]

Yes, some of my events contain all 19 MAC algorithms but not all. Some of these algorithms only occur occasionally in other events.

I try your search and it looks okay:

test3.png

[yuanliu]

---

@bnybln030 wrote:

Yes, some of my events contain all 19 MAC algorithms but not all. Some of these algorithms only occur occasionally in other events.

---

In that case, max_match=0 is your answer. (See "Optional arguments" in rex.)

 You need to post your complete search line; all screenshots only shows the regex portion, but not the complete rex command. (Also, post search line in text; screenshot is not a good way for other people to help.)

[ITWhisperer]

There is more to the search (you have counts) - can you share your complete search?

[bnybln030]

Here is  now a picture from field extration in splunk. It is the same result like my search....

test2.png

[ITWhisperer]

These appear to be counts of the occurrence of these algorithms in your events pipeline - perhaps only these 8 are present?