topic Re: ingesting nmap xml output in Splunk Search

ingesting nmap xml output

jnames10 — Thu, 07 Oct 2021 22:44:03 GMT

I have been trying to get nmap output into Splunk. I thought the xml output would be nice and straightforward!

Whilst the events are separated, the issue is having multiple values of the same field in an event.

automatic field extractions pick up the first <port... section but the rest are ignored. I tried using KV_MODE=xml but that didn't make a difference. I thought Splunk was quite happy pulling in multi values with xml but maybe its not quite the xml Splunk is expecting.

I am on Splunk Cloud so cli changes are not an option.

Any pointers appreciated! Thanks.

Re: ingesting nmap xml output

kamlesh_vaghela — Sat, 09 Oct 2021 15:16:24 GMT

@jnames10

Is it OK if you have a search which can extract fields for you? If yes can you please share inn which fields you are interested?

Re: ingesting nmap xml output

jnames10 — Sat, 09 Oct 2021 18:17:23 GMT

Hi Kamlesh,
Thanks for taking a look. I wondered if it might have to be search time extraction. The fields would be addr, name, portid, state

Ideally I want to end up with output I can go on to use in further searches, so if I had a table like below I could outputlookup it into a lookup table to use again.

addr	name	portid	state	portid	state	portid	state	and on for each portid value in event
81.123.123.123	host.name.resolved.com	21	filtered	22	open	23	open

or would I be better to have it like this?:

addr	name	portid	state
81.123.123.123	host.name.resolved.com	21	filtered
81.123.123.123	host.name.resolved.com	22	open
81.123.123.123	host.name.resolved.com	23	open
next ip	next name	etc etc

thanks again

Re: ingesting nmap xml output

PickleRick — Sat, 09 Oct 2021 18:37:36 GMT

Tried to replicate your problem. Created a simple test xml sourcetype, imported the event... et voila.

So there must be something wrong with your extraction. Are you sure you didn't create the extractions for this sourcetype in wrong app and forgot to give permissions?

Re: ingesting nmap xml output

kamlesh_vaghela — Sun, 10 Oct 2021 05:19:14 GMT

@jnames10

Can you please try this?

YOUR_SEARCH | spath | rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0 | eval t=mvzip(portid,state) | mvexpand t | eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1) | table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state | rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name

My Sample Search :

| makeresults | eval _raw="<host starttime=\"1633560153\" endtime=\"1633560291\"><status state=\"up\" reason=\"timestamp-reply\" reason_ttl=\"34\"/><address addr=\"81.123.123.123\" addrtype=\"ipv4\"/><hostnames><hostname name=\"host.name.resolved.com\" type=\"PTR\"/></hostnames><ports><port protocol=\"tcp\" portid=\"21\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ftp\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"22\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ssh\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"23\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"telnet\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"80\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"443\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"https\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8000\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-alt\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8080\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-proxy\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8888\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"sun-answerbook\" method=\"table\" conf=\"3\"/></port></ports><times srtt=\"15851\" rttvar=\"15851\" to=\"100000\"/></host>" | spath | rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0 | eval t=mvzip(portid,state) | mvexpand t | eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1) | table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state | rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name

Output:

Thanks
KV
▄︻̷̿┻̿═━一 😉

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

Re: ingesting nmap xml output

jnames10 — Mon, 11 Oct 2021 12:04:11 GMT

I am jealous! Can you share the source type settings or did you just accept the auto/defaults? any changes to line breaking or kv_mode? Thanks!

Re: ingesting nmap xml output

PickleRick — Tue, 12 Oct 2021 05:23:40 GMT

In fact all I did was add a KV_MODE entry. The rest was left as it was.

Re: ingesting nmap xml output

jnames10 — Tue, 12 Oct 2021 21:51:59 GMT

@kamlesh_vaghelathank you very much for that. i was very helpful! I am able to build some tables. It looks like Splunk Cloud is treating the XML input differently and not importing it in the same way as Splunk Enterprise. I am still fiddling to try and get it working but the regex you've done is really appreciated!

Thank you!

Re: ingesting nmap xml output

jnames10 — Tue, 12 Oct 2021 21:53:36 GMT

@PickleRickthanks for taking the time to do that. I fired up Splunk Enterprise and tried it and ok first time like you said. There is a complication around with my Splunk Cloud or just Splunk Cloud. Will do some more fiddling! Thanks again.

Re: ingesting nmap xml output

RMcCurdyDOTcom — Thu, 17 Aug 2023 19:02:41 GMT

XtremeNmapParser ! to convert the xml to JSON and then used HEC to send it all to Spunk!

https://github.com/xtormin/XtremeNmapParser/issues/1

Re: ingesting nmap xml output

RMcCurdyDOTcom — Tue, 26 Sep 2023 13:59:22 GMT

nmap XML to SPLUNK HEC !!!

https://github.com/freeload101/SCRIPTS/blob/b3f83288a9f289d86f6cdd04898478d0427097ce/Bash/NMAP_FRUIT.sh#L80

Re: ingesting nmap xml output

RMcCurdyDOTcom — Wed, 06 Mar 2024 18:13:24 GMT

got nasty gram for posting links

search online for freeload101 github in scripts nmap_fruit.sh