How to parse grepable Nmap output?

[rfiscus]

I have several events with similar to this raw data field that I would like to break down into a new event for each IP, port, etc:

Host: 172.30.x.x () Ports: 22/open/tcp//ssh///, 80/open/tcp//http///, 443/open/tcp//https///, 5000/open/tcp//upnp///, 5564/open/tcp/////, 5570/open/tcp/////, 5678/open/tcp//rrac///, 5988/open/tcp//wbem-http///, 5989/open/tcp//wbem-https///, 8008/open/tcp//http///, 8099/open/tcp//unknown///, 45454/open/tcp/////

So in my search (for this example) I want 12 lines that start with this IP address and in the second column have each of the ports listed respectively, followed by the columns port, status, proto, and desc.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw "(?i)Host:\s(?<dest_ip>\S+)\s+\(\)\s+Ports:\s+(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)"
| table _time dest_ip port status proto desc
| sort dest_ip

Currently my search works properly for the first port, but does not iterate through to create a new line for each consecutive port. I have read a lot about the makemv delim, but can't seem to make this work. Any ideas?

[rfiscus]

 index = ports_services sourcetype = nonwindows:ports Host Ports
 | rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
 | rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
 | rex field=_raw "OS:\s(?<os>\w+)"
 | eval os = if(isnull(os),"unknown",os)
 | eval mv=mvzip(port, status) 
 | eval mv=mvzip(mv, proto) 
 | eval mv=mvzip(mv, desc) 
 | mvexpand mv 
 | makemv mv delim="," 
 | eval ports=mvindex(mv, 0) 
 | eval status=mvindex(mv, 1)
 | eval proto=mvindex(mv, 2)
 | eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
 | table dest_ip ports status proto desc os
 | sort dest_ip

[sundareshr · Accepted solution]

This should give you an idea


| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" | table _time dest_ip ports | mvexpand ports


You should also look at mvzip command.

[rfiscus]

Finally got it, thank you for all your help!

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
| rex field=_raw "OS:\s(?<os>\w+)"
| eval os = if(isnull(os),"unknown",os)
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
| table dest_ip ports status proto desc os
| sort dest_ip

[rfiscus]

That gives me an error. Error in 'rex' command: Encountered the following error while compiling the regex '(?i)Host:\s(?\S+)': Regex: unrecognized character after (? or (?-

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" 
| table _time dest_ip ports 
| mvexpand ports

[rfiscus]

That does get me close, just now need to also separate the ports column.
-time dest_ip ports
2015-11-17 10:23:38 172.30.x.x 22/open/tcp//ssh///

2015-11-17 10:23:38 172.30.x.x 80/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 443/open/tcp//https///

2015-11-17 10:23:38 172.30.x.x 5000/open/tcp//upnp///

2015-11-17 10:23:38 172.30.x.x 5564/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5570/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5678/open/tcp//rrac///

2015-11-17 10:23:38 172.30.x.x 5988/open/tcp//wbem-http///

2015-11-17 10:23:38 172.30.x.x 5989/open/tcp//wbem-https///

2015-11-17 10:23:38 172.30.x.x 8008/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 8099/open/tcp//unknown///

2015-11-17 10:23:38 172.30.x.x 45454/open/tcp///

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Ports:\s+(?<ports>.+)\/\/"
| rex field=_raw max_match=50  (?i)Host:\s(?<dest_ip>\S+)
| makemv delim="," ports
| table _time dest_ip ports
| sort dest_ip
| mvexpand ports

[sundareshr]

Build on this

| rex field=s max_match=50 "Host:\s(?<dest_ip>\S+)" | rex field=s max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" | eval mv=mvzip(port, status) | eval mv=mvzip(mv, proto) | eval mv=mvzip(mv, desc) | mvexpand mv | makemv mv delim="," | eval ports=mvindex(mv, 0) | table dest_ip, ports

[rfiscus]

I am close, the only issue I am having is if there is not a desc, the event does not show up. I somehow have to capture the null value in the regex.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" 
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=mvindex(mv, 3)
| table dest_ip ports status proto desc
| sort dest_ip

[sundareshr]

I would modify the rex for desc to something like this ([\w+|\/]) (please test for correctness). OR ...| eval s=replace(s, "/////", "//unk///") | rex ...

Also, in the rex command, you could change the max_match to 0 (unlimited) vs 50

[rfiscus]

Sorry, I see, can I do the same thing for the / // /// delimitors?

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=_raw max_match=50 "Ports:\s+(?.+)"
| rex field=_raw max_match=50  (?i)Host:\s(?\S+)
| makemv delim="," ports
| table _time dest_ip ports 
| mvexpand ports

[rfiscus]

There will be variable number of ports, not always 12.

[rfiscus]

Search Output should look like this:

_time                                dest_ip           port     status     proto     desc
2015-11-17 10:23:38     172.30.x.x      22         open       tcp        ssh
2015-11-17 10:23:38     172.30.x.x      80         open       tcp        http
2015-11-17 10:23:38     172.30.x.x      443       open       tcp        https
2015-11-17 10:23:38     172.30.x.x      5000     open       tcp        upnp
2015-11-17 10:23:38     172.30.x.x      5564     open       tcp        
2015-11-17 10:23:38     172.30.x.x      5570     open       tcp        rrac

so on and so forth.

[RMcCurdyDOTcom]

I used XtremeNmapParser from github to convert the xml to JSON and then used HEC to send it all to Spunk!

https://github.com/xtormin/XtremeNmapParser/issues/1

[RMcCurdyDOTcom]

got nasty gram for posting links

search online for freeload101 github in scripts nmap_fruit.sh