topic Re: Field extraction with regex for Nmap result in Splunk Search

How do I extract this field with regex for Nmap result?

bnybln030 — Fri, 11 Mar 2022 15:54:57 GMT

Hi i want to extract the mac_algorithms field with regex from a nmap scan result. Does anyone have an idea how it works best? I've tried a few things, not all fields are found in Splunk.

Here you can see my example: https://regex101.com/r/eJ16fA/1

Here my nmap-scanning example:

kex_algorithms: (8)
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group-exchange-sha256
diffie-hellman-group-exchange-sha1
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
server_host_key_algorithms: (4)
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ssh-ed25519
encryption_algorithms: (14)
aes128-ctr
aes192-ctr
aes256-ctr
arcfour256
arcfour128
chacha20-poly1305@openssh.com
aes128-cbc
3des-cbc
blowfish-cbc
cast128-cbc
aes192-cbc
aes256-cbc
arcfour
rijndael-cbc@lysator.liu.se
mac_algorithms: (19)
hmac-md5-etm@openssh.com
hmac-sha1-etm@openssh.com
umac-64-etm@openssh.com
umac-128-etm@openssh.com
hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com
hmac-ripemd160-etm@openssh.com
hmac-sha1-96-etm@openssh.com
hmac-md5-96-etm@openssh.com
hmac-md5
hmac-sha1
umac-64@openssh.com
umac-128@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-ripemd160
hmac-ripemd160@openssh.com
hmac-sha1-96
hmac-md5-96
compression_algorithms: (2)
none
zlib@openssh.com"

Re: Field extraction with regex for Nmap result

ITWhisperer — Fri, 11 Mar 2022 08:54:06 GMT

I am not sure what the issue is here - your regex101 example seems to show all 19 algorithms have been matched. What is your expected/required result?

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 09:01:58 GMT

The problem is that not all values are found in splunk. I only see 8 but in the example on regex101 there are 19 different ones.

Here you can see my result:

Re: Field extraction with regex for Nmap result

ITWhisperer — Fri, 11 Mar 2022 09:15:26 GMT

There is more to the search (you have counts) - can you share your complete search?

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 09:25:06 GMT

Here is now a picture from field extration in splunk. It is the same result like my search....

Re: Field extraction with regex for Nmap result

yuanliu — Fri, 11 Mar 2022 10:11:43 GMT

Have you tested that your events contain all 19 MAC algorithms? (This can be tested by searching for those that you believe are missed by Splunk search.)
Is it possible that MAC may appear without leading space (hence not matching "^\s+") or appear after non-whitespace characters? MAC algorithm patterns are unique enough that you can simply use "\b" to delimit it. This example illustrates that Splunk does extract all 19 algos if events contain them all.

| makeresults | eval event=split(" kex_algorithms: (8) curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1 diffie-hellman-group1-sha1 server_host_key_algorithms: (4) ssh-rsa ssh-dss ecdsa-sha2-nistp256 ssh-ed25519 encryption_algorithms: (14) aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 chacha20-poly1305@openssh.com aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour rijndael-cbc@lysator.liu.se mac_algorithms: (19) hmac-md5-etm@openssh.com hmac-sha1-etm@openssh.com umac-64-etm@openssh.com umac-128-etm@openssh.com hmac-sha2-256-etm@openssh.com hmac-sha2-512-etm@openssh.com hmac-ripemd160-etm@openssh.com hmac-sha1-96-etm@openssh.com hmac-md5-96-etm@openssh.com hmac-md5 hmac-sha1 umac-64@openssh.com umac-128@openssh.com hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 hmac-ripemd160@openssh.com hmac-sha1-96 hmac-md5-96 compression_algorithms: (2) none zlib@openssh.com", " ") | mvexpand event ``` event simulation above ``` | rex field=event max_match=0 "\b(?<test>[uh]mac-[\w\-\.@]+)"

Output from the simulation is

event	test
kex_algorithms: (8)
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group-exchange-sha256
diffie-hellman-group-exchange-sha1
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
server_host_key_algorithms: (4)
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ssh-ed25519
encryption_algorithms: (14)
aes128-ctr
aes192-ctr
aes256-ctr
arcfour256
arcfour128
chacha20-poly1305@openssh.com
aes128-cbc
3des-cbc
blowfish-cbc
cast128-cbc
aes192-cbc
aes256-cbc
arcfour
rijndael-cbc@lysator.liu.se
mac_algorithms: (19)
hmac-md5-etm@openssh.com	hmac-md5-etm@openssh.com
hmac-sha1-etm@openssh.com	hmac-sha1-etm@openssh.com
umac-64-etm@openssh.com	umac-64-etm@openssh.com
umac-128-etm@openssh.com	umac-128-etm@openssh.com
hmac-sha2-256-etm@openssh.com	hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com	hmac-sha2-512-etm@openssh.com
hmac-ripemd160-etm@openssh.com	hmac-ripemd160-etm@openssh.com
hmac-sha1-96-etm@openssh.com	hmac-sha1-96-etm@openssh.com
hmac-md5-96-etm@openssh.com	hmac-md5-96-etm@openssh.com
hmac-md5	hmac-md5
hmac-sha1	hmac-sha1
umac-64@openssh.com	umac-64@openssh.com
umac-128@openssh.com	umac-128@openssh.com
hmac-sha2-256	hmac-sha2-256
hmac-sha2-512	hmac-sha2-512
hmac-ripemd160	hmac-ripemd160
hmac-ripemd160@openssh.com	hmac-ripemd160@openssh.com
hmac-sha1-96	hmac-sha1-96
hmac-md5-96	hmac-md5-96
compression_algorithms: (2)
none
zlib@openssh.com

Note

"^\s+\K" is really just "^\s+".
An explicit "-" after "(hmac|umac)" better expresses the MAC format, thus reduces potential false positives.

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 10:34:42 GMT

Yes, some of my events contain all 19 MAC algorithms but not all. Some of these algorithms only occur occasionally in other events.

I try your search and it looks okay:

Re: Field extraction with regex for Nmap result

ITWhisperer — Fri, 11 Mar 2022 10:41:08 GMT

These appear to be counts of the occurrence of these algorithms in your events pipeline - perhaps only these 8 are present?

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 10:57:28 GMT

I see in the field extractor that only the first entry is recognized....

Re: Field extraction with regex for Nmap result

ITWhisperer — Fri, 11 Mar 2022 11:00:28 GMT

You probably need to enable multivalue field extraction?

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 11:03:46 GMT

How can i do it?

Re: Field extraction with regex for Nmap result

yuanliu — Fri, 11 Mar 2022 15:51:20 GMT

@bnybln030 wrote:
I see in the field extractor that only the first entry is recognized....

Can you post the complete filter line in your search? (From "|" all the way to the end or next "|") The screenshot only captures "\b(?<test>[hu]mac-[\w\-\.@]+)", which doesn't contain a valid SPL command. My posted filter is

| rex field=event max_match=0 "\b(?<test>[hu]mac-[\w\-\.@]+)"

The argument "max_match=0" makes sure to match all entries in a multi-line event. Side note: If your input is as that screenshot shows, containing multiple lines of MAC algorithms, literally copying that regex101 into rex command without max_match=0 will only extract the first occurrence. This could explain why you only got 8 in the first place. (See "Optional arguments" in rex.

Here is a test with all 19 algos in one event:

With max_match=0

| makeresults | eval event=" kex_algorithms: (8) curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1 diffie-hellman-group1-sha1 server_host_key_algorithms: (4) ssh-rsa ssh-dss ecdsa-sha2-nistp256 ssh-ed25519 encryption_algorithms: (14) aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 chacha20-poly1305@openssh.com aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour rijndael-cbc@lysator.liu.se mac_algorithms: (19) hmac-md5-etm@openssh.com hmac-sha1-etm@openssh.com umac-64-etm@openssh.com umac-128-etm@openssh.com hmac-sha2-256-etm@openssh.com hmac-sha2-512-etm@openssh.com hmac-ripemd160-etm@openssh.com hmac-sha1-96-etm@openssh.com hmac-md5-96-etm@openssh.com hmac-md5 hmac-sha1 umac-64@openssh.com umac-128@openssh.com hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 hmac-ripemd160@openssh.com hmac-sha1-96 hmac-md5-96 compression_algorithms: (2) none zlib@openssh.com" ``` event simulation above ``` | rex field=event max_match=0 "\b(?<test>[hu]mac-[\w\-\.@]+)" | eval algocount=mvcount(test) | table algocount event test

algocount

event

test

  kex_algorithms: (8)
      curve25519-sha256@libssh.org
      ecdh-sha2-nistp256
      ecdh-sha2-nistp384
      ecdh-sha2-nistp521
      diffie-hellman-group-exchange-sha256
      diffie-hellman-group-exchange-sha1
      diffie-hellman-group14-sha1
      diffie-hellman-group1-sha1
  server_host_key_algorithms: (4)
      ssh-rsa
      ssh-dss
      ecdsa-sha2-nistp256
      ssh-ed25519
  encryption_algorithms: (14)
      aes128-ctr
      aes192-ctr
      aes256-ctr
      arcfour256
      arcfour128
      chacha20-poly1305@openssh.com
      aes128-cbc
      3des-cbc
      blowfish-cbc
      cast128-cbc
      aes192-cbc
      aes256-cbc
      arcfour
      rijndael-cbc@lysator.liu.se
  mac_algorithms: (19)
      hmac-md5-etm@openssh.com
      hmac-sha1-etm@openssh.com
      umac-64-etm@openssh.com
      umac-128-etm@openssh.com
      hmac-sha2-256-etm@openssh.com
      hmac-sha2-512-etm@openssh.com
      hmac-ripemd160-etm@openssh.com
      hmac-sha1-96-etm@openssh.com
      hmac-md5-96-etm@openssh.com
      hmac-md5
      hmac-sha1
      umac-64@openssh.com
      umac-128@openssh.com
      hmac-sha2-256
      hmac-sha2-512
      hmac-ripemd160
      hmac-ripemd160@openssh.com
      hmac-sha1-96
      hmac-md5-96
  compression_algorithms: (2)
      none
      zlib@openssh.com

hmac-md5-etm@openssh.com
hmac-sha1-etm@openssh.com
umac-64-etm@openssh.com
umac-128-etm@openssh.com
hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com
hmac-ripemd160-etm@openssh.com
hmac-sha1-96-etm@openssh.com
hmac-md5-96-etm@openssh.com
hmac-md5
hmac-sha1
umac-64@openssh.com
umac-128@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-ripemd160
hmac-ripemd160@openssh.com
hmac-sha1-96
hmac-md5-96

No max_match=0

| makeresults | eval event=" kex_algorithms: (8) curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group-exchange-sha1 diffie-hellman-group14-sha1 diffie-hellman-group1-sha1 server_host_key_algorithms: (4) ssh-rsa ssh-dss ecdsa-sha2-nistp256 ssh-ed25519 encryption_algorithms: (14) aes128-ctr aes192-ctr aes256-ctr arcfour256 arcfour128 chacha20-poly1305@openssh.com aes128-cbc 3des-cbc blowfish-cbc cast128-cbc aes192-cbc aes256-cbc arcfour rijndael-cbc@lysator.liu.se mac_algorithms: (19) hmac-md5-etm@openssh.com hmac-sha1-etm@openssh.com umac-64-etm@openssh.com umac-128-etm@openssh.com hmac-sha2-256-etm@openssh.com hmac-sha2-512-etm@openssh.com hmac-ripemd160-etm@openssh.com hmac-sha1-96-etm@openssh.com hmac-md5-96-etm@openssh.com hmac-md5 hmac-sha1 umac-64@openssh.com umac-128@openssh.com hmac-sha2-256 hmac-sha2-512 hmac-ripemd160 hmac-ripemd160@openssh.com hmac-sha1-96 hmac-md5-96 compression_algorithms: (2) none zlib@openssh.com" ``` event simulation above ``` | rex field=event "\b(?<test>[hu]mac-[\w\-\.@]+)" | eval algocount=mvcount(test) | table algocount event test

algocount

event

test

  kex_algorithms: (8)
      curve25519-sha256@libssh.org
      ecdh-sha2-nistp256
      ecdh-sha2-nistp384
      ecdh-sha2-nistp521
      diffie-hellman-group-exchange-sha256
      diffie-hellman-group-exchange-sha1
      diffie-hellman-group14-sha1
      diffie-hellman-group1-sha1
  server_host_key_algorithms: (4)
      ssh-rsa
      ssh-dss
      ecdsa-sha2-nistp256
      ssh-ed25519
  encryption_algorithms: (14)
      aes128-ctr
      aes192-ctr
      aes256-ctr
      arcfour256
      arcfour128
      chacha20-poly1305@openssh.com
      aes128-cbc
      3des-cbc
      blowfish-cbc
      cast128-cbc
      aes192-cbc
      aes256-cbc
      arcfour
      rijndael-cbc@lysator.liu.se
  mac_algorithms: (19)
      hmac-md5-etm@openssh.com
      hmac-sha1-etm@openssh.com
      umac-64-etm@openssh.com
      umac-128-etm@openssh.com
      hmac-sha2-256-etm@openssh.com
      hmac-sha2-512-etm@openssh.com
      hmac-ripemd160-etm@openssh.com
      hmac-sha1-96-etm@openssh.com
      hmac-md5-96-etm@openssh.com
      hmac-md5
      hmac-sha1
      umac-64@openssh.com
      umac-128@openssh.com
      hmac-sha2-256
      hmac-sha2-512
      hmac-ripemd160
      hmac-ripemd160@openssh.com
      hmac-sha1-96
      hmac-md5-96
  compression_algorithms: (2)
      none
      zlib@openssh.com

hmac-md5-etm@openssh.com

Re: Field extraction with regex for Nmap result

yuanliu — Fri, 11 Mar 2022 15:59:00 GMT

@bnybln030 wrote:
Yes, some of my events contain all 19 MAC algorithms but not all. Some of these algorithms only occur occasionally in other events.

In that case, max_match=0 is your answer. (See "Optional arguments" in rex.)

You need to post your complete search line; all screenshots only shows the regex portion, but not the complete rex command. (Also, post search line in text; screenshot is not a good way for other people to help.)

Re: Field extraction with regex for Nmap result

bnybln030 — Fri, 11 Mar 2022 17:56:15 GMT

You are right, on my screenshot you see the complete search. With "max_match=0" i get the right results:

If I dont set "max_match=0" then I get exactly the same result as with field extractor: