Splunk Search

How to parse grepable Nmap output?

rfiscus
Path Finder

I have several events with similar to this raw data field that I would like to break down into a new event for each IP, port, etc:

Host: 172.30.x.x () Ports: 22/open/tcp//ssh///, 80/open/tcp//http///, 443/open/tcp//https///, 5000/open/tcp//upnp///, 5564/open/tcp/////, 5570/open/tcp/////, 5678/open/tcp//rrac///, 5988/open/tcp//wbem-http///, 5989/open/tcp//wbem-https///, 8008/open/tcp//http///, 8099/open/tcp//unknown///, 45454/open/tcp/////

So in my search (for this example) I want 12 lines that start with this IP address and in the second column have each of the ports listed respectively, followed by the columns port, status, proto, and desc.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw "(?i)Host:\s(?<dest_ip>\S+)\s+\(\)\s+Ports:\s+(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)"
| table _time dest_ip port status proto desc
| sort dest_ip

Currently my search works properly for the first port, but does not iterate through to create a new line for each consecutive port. I have read a lot about the makemv delim, but can't seem to make this work. Any ideas?

0 Karma
1 Solution

sundareshr
Legend

This should give you an idea


| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" | table _time dest_ip ports | mvexpand ports

You should also look at mvzip command.

View solution in original post

rfiscus
Path Finder
 index = ports_services sourcetype = nonwindows:ports Host Ports
 | rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
 | rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
 | rex field=_raw "OS:\s(?<os>\w+)"
 | eval os = if(isnull(os),"unknown",os)
 | eval mv=mvzip(port, status) 
 | eval mv=mvzip(mv, proto) 
 | eval mv=mvzip(mv, desc) 
 | mvexpand mv 
 | makemv mv delim="," 
 | eval ports=mvindex(mv, 0) 
 | eval status=mvindex(mv, 1)
 | eval proto=mvindex(mv, 2)
 | eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
 | table dest_ip ports status proto desc os
 | sort dest_ip
0 Karma

sundareshr
Legend

This should give you an idea


| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" | table _time dest_ip ports | mvexpand ports

You should also look at mvzip command.

rfiscus
Path Finder

Finally got it, thank you for all your help!

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/+(?<status>\w+)\/+(?<proto>\w+)\/+(?<desc>\w+|\/)"
| rex field=_raw "OS:\s(?<os>\w+)"
| eval os = if(isnull(os),"unknown",os)
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=if(mvindex(mv, 3) == "/","null",mvindex(mv,3))
| table dest_ip ports status proto desc os
| sort dest_ip
0 Karma

rfiscus
Path Finder

That gives me an error. Error in 'rex' command: Encountered the following error while compiling the regex '(?i)Host:\s(?\S+)': Regex: unrecognized character after (? or (?-

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=s max_match=50 "(?i)Host:\s(?\S+)"
| rex field=s max_match=50 "[Ports:|,]\s(?\d+)" 
| table _time dest_ip ports 
| mvexpand ports
0 Karma

rfiscus
Path Finder

That does get me close, just now need to also separate the ports column.
-time dest_ip ports
2015-11-17 10:23:38 172.30.x.x 22/open/tcp//ssh///

2015-11-17 10:23:38 172.30.x.x 80/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 443/open/tcp//https///

2015-11-17 10:23:38 172.30.x.x 5000/open/tcp//upnp///

2015-11-17 10:23:38 172.30.x.x 5564/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5570/open/tcp/////

2015-11-17 10:23:38 172.30.x.x 5678/open/tcp//rrac///

2015-11-17 10:23:38 172.30.x.x 5988/open/tcp//wbem-http///

2015-11-17 10:23:38 172.30.x.x 5989/open/tcp//wbem-https///

2015-11-17 10:23:38 172.30.x.x 8008/open/tcp//http///

2015-11-17 10:23:38 172.30.x.x 8099/open/tcp//unknown///

2015-11-17 10:23:38 172.30.x.x 45454/open/tcp///

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Ports:\s+(?<ports>.+)\/\/"
| rex field=_raw max_match=50  (?i)Host:\s(?<dest_ip>\S+)
| makemv delim="," ports
| table _time dest_ip ports
| sort dest_ip
| mvexpand ports
0 Karma

sundareshr
Legend

Build on this

| rex field=s max_match=50 "Host:\s(?<dest_ip>\S+)" | rex field=s max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" | eval mv=mvzip(port, status) | eval mv=mvzip(mv, proto) | eval mv=mvzip(mv, desc) | mvexpand mv | makemv mv delim="," | eval ports=mvindex(mv, 0) | table dest_ip, ports
0 Karma

rfiscus
Path Finder

I am close, the only issue I am having is if there is not a desc, the event does not show up. I somehow have to capture the null value in the regex.

index = ports_services sourcetype = nonwindows:ports Host Ports
| rex field=_raw max_match=50 "Host:\s(?<dest_ip>\S+)" 
| rex field=_raw max_match=50 "[Ports:|,]\s?(?<port>\d+)\/(?<status>\w+)\/(?<proto>\w+)\/\/(?<desc>\w+)" 
| eval mv=mvzip(port, status) 
| eval mv=mvzip(mv, proto) 
| eval mv=mvzip(mv, desc) 
| mvexpand mv 
| makemv mv delim="," 
| eval ports=mvindex(mv, 0) 
| eval status=mvindex(mv, 1)
| eval proto=mvindex(mv, 2)
| eval desc=mvindex(mv, 3)
| table dest_ip ports status proto desc
| sort dest_ip
0 Karma

sundareshr
Legend

I would modify the rex for desc to something like this ([\w+|\/]) (please test for correctness). OR ...| eval s=replace(s, "/////", "//unk///") | rex ...

Also, in the rex command, you could change the max_match to 0 (unlimited) vs 50

0 Karma

rfiscus
Path Finder

Sorry, I see, can I do the same thing for the / // /// delimitors?

index = ports_services sourcetype = nonwindows:ports Host Ports

| rex field=_raw max_match=50 "Ports:\s+(?.+)"
| rex field=_raw max_match=50  (?i)Host:\s(?\S+)
| makemv delim="," ports
| table _time dest_ip ports 
| mvexpand ports
0 Karma

rfiscus
Path Finder

There will be variable number of ports, not always 12.

0 Karma

rfiscus
Path Finder

Search Output should look like this:

_time                                dest_ip           port     status     proto     desc
2015-11-17 10:23:38     172.30.x.x      22         open       tcp        ssh
2015-11-17 10:23:38     172.30.x.x      80         open       tcp        http
2015-11-17 10:23:38     172.30.x.x      443       open       tcp        https
2015-11-17 10:23:38     172.30.x.x      5000     open       tcp        upnp
2015-11-17 10:23:38     172.30.x.x      5564     open       tcp        
2015-11-17 10:23:38     172.30.x.x      5570     open       tcp        rrac

so on and so forth.

0 Karma

RMcCurdyDOTcom
Explorer

I used XtremeNmapParser from github to convert the xml to JSON and then used HEC to send it all to Spunk!

https://github.com/xtormin/XtremeNmapParser/issues/1

0 Karma

RMcCurdyDOTcom
Explorer

got nasty gram for posting links

search online for freeload101 github in scripts nmap_fruit.sh

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...