Splunk Search

How to parse grepable Nmap output?

Path Finder

I have several events with similar to this raw data field that I would like to break down into a new event for each IP, port, etc:

Host: 172.30.x.x () Ports: 22/open/tcp//ssh///, 80/open/tcp//http///, 443/open/tcp//https///, 5000/open/tcp//upnp///, 5564/open/tcp/////, 5570/open/tcp/////, 5678/open/tcp//rrac///, 5988/open/tcp//wbem-http///, 5989/open/tcp//wbem-https///, 8008/open/tcp//http///, 8099/open/tcp//unknown///, 45454/open/tcp/////

So in my search (for this example) I want 12 lines that start with this IP address and in the second column have each of the ports listed respectively, followed by the columns port, status, proto, and desc.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw "(?i)Host:\s(?<dest_ip>\S+)\s+\(\)\s+Ports:\s+(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)"
| table _time dest_ip port status proto desc
| sort dest_ip

Currently my search works properly for the first port, but does not iterate through to create a new line for each consecutive port. I have read a lot about the makemv delim, but can't seem to make this work. Any ideas?

0 Karma
1 Solution

Legend

This should give you an idea


| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" | table _time dest_ip ports | mvexpand ports

You should also look at mvzip command.

View solution in original post

Path Finder
 index = ports_services sourcetype = nonwindows:ports Host Ports
 | rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
 | rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
 | rex field=_raw "OS:\s(?<os>\w+)"
 | eval os = if(isnull(os),"unknown",os)
 | eval mv=mvzip(port, status) 
 | eval mv=mvzip(mv, proto) 
 | eval mv=mvzip(mv, desc) 
 | mvexpand mv 
 | makemv mv delim="," 
 | eval ports=mvindex(mv, 0) 
 | eval status=mvindex(mv, 1)
 | eval proto=mvindex(mv, 2)
 | eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
 | table dest_ip ports status proto desc os
 | sort dest_ip
0 Karma

Legend

This should give you an idea


| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" | table _time dest_ip ports | mvexpand ports

You should also look at mvzip command.

View solution in original post

Path Finder

Finally got it, thank you for all your help!

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
| rex field=_raw "OS:\s(?<os>\w+)"
| eval os = if(isnull(os),"unknown",os)
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
| table dest_ip ports status proto desc os
| sort dest_ip
0 Karma

Path Finder

That gives me an error. Error in 'rex' command: Encountered the following error while compiling the regex '(?i)Host:\s(?\S+)': Regex: unrecognized character after (? or (?-

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" 
| table _time dest_ip ports 
| mvexpand ports
0 Karma

Path Finder

That does get me close, just now need to also separate the ports column.
-time dest_ip ports
2015-11-17 10:23:38 172.30.x.x 22/open/tcp//ssh///

2015-11-17 10:23:38 172.30.x.x 80/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 443/open/tcp//https///

2015-11-17 10:23:38 172.30.x.x 5000/open/tcp//upnp///

2015-11-17 10:23:38 172.30.x.x 5564/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5570/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5678/open/tcp//rrac///

2015-11-17 10:23:38 172.30.x.x 5988/open/tcp//wbem-http///

2015-11-17 10:23:38 172.30.x.x 5989/open/tcp//wbem-https///

2015-11-17 10:23:38 172.30.x.x 8008/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 8099/open/tcp//unknown///

2015-11-17 10:23:38 172.30.x.x 45454/open/tcp///

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Ports:\s+(?<ports>.+)\/\/"
| rex field=_raw max_match=50  (?i)Host:\s(?<dest_ip>\S+)
| makemv delim="," ports
| table _time dest_ip ports
| sort dest_ip
| mvexpand ports
0 Karma

Legend

Build on this

| rex field=s max_match=50 "Host:\s(?<dest_ip>\S+)" | rex field=s max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" | eval mv=mvzip(port, status) | eval mv=mvzip(mv, proto) | eval mv=mvzip(mv, desc) | mvexpand mv | makemv mv delim="," | eval ports=mvindex(mv, 0) | table dest_ip, ports
0 Karma

Path Finder

I am close, the only issue I am having is if there is not a desc, the event does not show up. I somehow have to capture the null value in the regex.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" 
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=mvindex(mv, 3)
| table dest_ip ports status proto desc
| sort dest_ip
0 Karma

Legend

I would modify the rex for desc to something like this ([\w+|\/]) (please test for correctness). OR ...| eval s=replace(s, "/////", "//unk///") | rex ...

Also, in the rex command, you could change the max_match to 0 (unlimited) vs 50

0 Karma

Path Finder

Sorry, I see, can I do the same thing for the / // /// delimitors?

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=_raw max_match=50 "Ports:\s+(?.+)"
| rex field=_raw max_match=50  (?i)Host:\s(?\S+)
| makemv delim="," ports
| table _time dest_ip ports 
| mvexpand ports
0 Karma

Path Finder

There will be variable number of ports, not always 12.

0 Karma

Path Finder

Search Output should look like this:

_time                                dest_ip           port     status     proto     desc
2015-11-17 10:23:38     172.30.x.x      22         open       tcp        ssh
2015-11-17 10:23:38     172.30.x.x      80         open       tcp        http
2015-11-17 10:23:38     172.30.x.x      443       open       tcp        https
2015-11-17 10:23:38     172.30.x.x      5000     open       tcp        upnp
2015-11-17 10:23:38     172.30.x.x      5564     open       tcp        
2015-11-17 10:23:38     172.30.x.x      5570     open       tcp        rrac

so on and so forth.

0 Karma