Okay that was a headache, but satisfying nonetheless - in hindsight (as it always is), it was actually much more straightforward than then numerous avenues I looked into.
I was able to extract all services, ports, daemons and banners using the following setup below.
In addition, I found it useful to separate out subdomains also. Unfortunately the regexes will not work for all domains/subdomains, but YMMV.
/path/to/greppable/nmap/output.gnmap
inputs.conf
[monitor:///path/to/greppable/nmap/*.gnmap]
index = nmap
sourcetype = nmap
queue = parsingQueue
disabled = 0
props.conf
[nmap]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-nmap = NMAPsetnull,NMAPsetparsing
EXTRACT-ip = (?i)Host: (?P<ip>[^ ]+)
EXTRACT-hostname = (?i)^[^\(]*\((?P<hostname>[^\)]+)
EXTRACT-subdomain = (?i)\(.*?\.(?P<subdomain>\w+\.\w+\.\w+\.\w+)(?=\))
EXTRACT-domain = (?i)\..*?\.(?P<domain>\w+\.\w+\.\w+)(?=\))
REPORT-ports = ports
transforms.conf
[NMAPsetnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[NMAPsetparsing]
REGEX = Ports:
DEST_KEY = queue
FORMAT = indexQueue
[ports]
REGEX = \s(?<port>\d+)/(?<state>[^/]+)/(?<proto>[^/]+)//(?<daemon>[^/]*)//(?<banner>[^/]*)/
DEFAULT_VALUE = null
MV_ADD = TRUE
One specific note regarding greppable NMAP output that you should take care with. A very small number of services discovered by NMAP and dumped into greppable NMAP output are formatted incorrectly. e.g.:
2049/open/tcp//nfs (nfs V2-4)/(nfs:100003*2-4)/2-4 (rpc #100003)/
111/open/tcp//rpcbind/N//
It's enough to skew your results.
The backslashes are in the incorrect spot when compared with all the other NMAP discovered services.
The format should be:
2049/open/tcp//nfs (nfs V2-4)//(nfs:100003*2-4)/2-4 (rpc #100003)/
111/open/tcp//rpcbind//N/
If your network does have services that are formatted by NMAP like this, you may wish to run a find/replace over your gnmap files something like follows:
sed -i 's"2049\/open\/tcp\/\/nfs "2049\/open\/tcp\/\/nfs\//"pg' *.gnmap
sed -i 's"111\/open\/tcp\/\/rpcbind\/N\/\/"111\/open\/tcp\/\/rpcbind\/\/N\/"pg' *.gnmap
sed -i 's"111\/open\/tcp\/\/rpcbind\s"111\/open\/tcp\/\/rpcbind\/\/"pg' *.gnmap
Which corrects the three anomalous services I discovered on the networks I look at. You may find more (drop a note here!) or you may need to adjust those regexes for sed to parse them properly.
... View more
