topic Re: How to parse grepable Nmap output? in Splunk Search

How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 19:13:18 GMT

I have several events with similar to this raw data field that I would like to break down into a new event for each IP, port, etc:

Host: 172.30.x.x () Ports: 22/open/tcp//ssh///, 80/open/tcp//http///, 443/open/tcp//https///, 5000/open/tcp//upnp///, 5564/open/tcp/////, 5570/open/tcp/////, 5678/open/tcp//rrac///, 5988/open/tcp//wbem-http///, 5989/open/tcp//wbem-https///, 8008/open/tcp//http///, 8099/open/tcp//unknown///, 45454/open/tcp/////

So in my search (for this example) I want 12 lines that start with this IP address and in the second column have each of the ports listed respectively, followed by the columns port, status, proto, and desc.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw "(?i)Host:\s(?<dest_ip>\S+)\s+\(\)\s+Ports:\s+(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)"
| table _time dest_ip port status proto desc
| sort dest_ip

Currently my search works properly for the first port, but does not iterate through to create a new line for each consecutive port. I have read a lot about the makemv delim, but can't seem to make this work. Any ideas?

Re: How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 19:18:27 GMT

Search Output should look like this:

_time                                dest_ip           port     status     proto     desc
2015-11-17 10:23:38     172.30.x.x      22         open       tcp        ssh
2015-11-17 10:23:38     172.30.x.x      80         open       tcp        http
2015-11-17 10:23:38     172.30.x.x      443       open       tcp        https
2015-11-17 10:23:38     172.30.x.x      5000     open       tcp        upnp
2015-11-17 10:23:38     172.30.x.x      5564     open       tcp        
2015-11-17 10:23:38     172.30.x.x      5570     open       tcp        rrac

so on and so forth.

Re: How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 19:20:31 GMT

There will be variable number of ports, not always 12.

Re: How to parse grepable Nmap output?

sundareshr — Tue, 17 Nov 2015 19:54:34 GMT

This should give you an idea

You should also look at mvzip command.

Re: How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 20:46:34 GMT

That gives me an error. Error in 'rex' command: Encountered the following error while compiling the regex '(?i)Host:\s(?\S+)': Regex: unrecognized character after (? or (?-

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" 
| table _time dest_ip ports 
| mvexpand ports

Re: How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 20:54:28 GMT

Sorry, I see, can I do the same thing for the / // /// delimitors?

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=_raw max_match=50 "Ports:\s+(?.+)"
| rex field=_raw max_match=50  (?i)Host:\s(?\S+)
| makemv delim="," ports
| table _time dest_ip ports 
| mvexpand ports

Re: How to parse grepable Nmap output?

rfiscus — Tue, 17 Nov 2015 22:00:46 GMT

That does get me close, just now need to also separate the ports column.
-time dest_ip ports
2015-11-17 10:23:38 172.30.x.x 22/open/tcp//ssh///

2015-11-17 10:23:38 172.30.x.x 80/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 443/open/tcp//https///

2015-11-17 10:23:38 172.30.x.x 5000/open/tcp//upnp///

2015-11-17 10:23:38 172.30.x.x 5564/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5570/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5678/open/tcp//rrac///

2015-11-17 10:23:38 172.30.x.x 5988/open/tcp//wbem-http///

2015-11-17 10:23:38 172.30.x.x 5989/open/tcp//wbem-https///

2015-11-17 10:23:38 172.30.x.x 8008/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 8099/open/tcp//unknown///

2015-11-17 10:23:38 172.30.x.x 45454/open/tcp///

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Ports:\s+(?<ports>.+)\/\/"
| rex field=_raw max_match=50  (?i)Host:\s(?<dest_ip>\S+)
| makemv delim="," ports
| table _time dest_ip ports
| sort dest_ip
| mvexpand ports

Re: How to parse grepable Nmap output?

sundareshr — Wed, 18 Nov 2015 01:35:11 GMT

Build on this

| rex field=s max_match=50 "Host:\s(?<dest_ip>\S+)" | rex field=s max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" | eval mv=mvzip(port, status) | eval mv=mvzip(mv, proto) | eval mv=mvzip(mv, desc) | mvexpand mv | makemv mv delim="," | eval ports=mvindex(mv, 0) | table dest_ip, ports

Re: How to parse grepable Nmap output?

rfiscus — Wed, 18 Nov 2015 12:53:14 GMT

I am close, the only issue I am having is if there is not a desc, the event does not show up. I somehow have to capture the null value in the regex.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" 
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=mvindex(mv, 3)
| table dest_ip ports status proto desc
| sort dest_ip

Re: How to parse grepable Nmap output?

rfiscus — Wed, 18 Nov 2015 14:18:31 GMT

Finally got it, thank you for all your help!

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
| rex field=_raw "OS:\s(?<os>\w+)"
| eval os = if(isnull(os),"unknown",os)
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
| table dest_ip ports status proto desc os
| sort dest_ip

Re: How to parse grepable Nmap output?

sundareshr — Wed, 18 Nov 2015 14:27:31 GMT

I would modify the rex for desc to something like this ([\w+|\/]) (please test for correctness). OR ...| eval s=replace(s, "/////", "//unk///") | rex ...

Also, in the rex command, you could change the max_match to 0 (unlimited) vs 50

Re: How to parse grepable Nmap output?

rfiscus — Wed, 18 Nov 2015 14:45:17 GMT

 index = ports_services sourcetype = nonwindows:ports Host Ports
 | rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
 | rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
 | rex field=_raw "OS:\s(?<os>\w+)"
 | eval os = if(isnull(os),"unknown",os)
 | eval mv=mvzip(port, status) 
 | eval mv=mvzip(mv, proto) 
 | eval mv=mvzip(mv, desc) 
 | mvexpand mv 
 | makemv mv delim="," 
 | eval ports=mvindex(mv, 0) 
 | eval status=mvindex(mv, 1)
 | eval proto=mvindex(mv, 2)
 | eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
 | table dest_ip ports status proto desc os
 | sort dest_ip

Re: How to parse grepable Nmap output?

RMcCurdyDOTcom — Thu, 17 Aug 2023 19:02:30 GMT

I used XtremeNmapParser from github to convert the xml to JSON and then used HEC to send it all to Spunk!

https://github.com/xtormin/XtremeNmapParser/issues/1

Re: How to parse grepable Nmap output?

RMcCurdyDOTcom — Wed, 06 Mar 2024 18:12:54 GMT

got nasty gram for posting links

search online for freeload101 github in scripts nmap_fruit.sh