Splunk Search

ingesting nmap xml output

jnames10
Explorer

I have been trying to get nmap output into Splunk. I thought the xml output would be nice and straightforward!

Whilst the events are separated, the issue is having multiple values of the same field in an event.

automatic field extractions pick up the first <port... section but the rest are ignored. I tried using KV_MODE=xml but that didn't make a difference. I thought Splunk was quite happy pulling in multi values with xml but maybe its not quite the xml Splunk is expecting.

I am on Splunk Cloud so cli changes are not an option.

Any pointers appreciated! Thanks. 

Search___Splunk_8_2_2107.png

 

 

<host starttime="1633560153" endtime="1633560291"><status state="up" reason="timestamp-reply" reason_ttl="34"/>
<address addr="81.123.123.123" addrtype="ipv4"/>
<hostnames>
<hostname name="host.name.resolved.com" type="PTR"/>
</hostnames>
<ports><port protocol="tcp" portid="21"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="ftp" method="table" conf="3"/></port>
<port protocol="tcp" portid="22"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="ssh" method="table" conf="3"/></port>
<port protocol="tcp" portid="23"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="telnet" method="table" conf="3"/></port>
<port protocol="tcp" portid="80"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http" method="table" conf="3"/></port>
<port protocol="tcp" portid="443"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="https" method="table" conf="3"/></port>
<port protocol="tcp" portid="8000"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http-alt" method="table" conf="3"/></port>
<port protocol="tcp" portid="8080"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http-proxy" method="table" conf="3"/></port>
<port protocol="tcp" portid="8888"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="sun-answerbook" method="table" conf="3"/></port>
</ports>
<times srtt="15851" rttvar="15851" to="100000"/>
</host>

 

 

Labels (2)

PickleRick
SplunkTrust
SplunkTrust

Tried to replicate your problem. Created a simple test xml sourcetype, imported the event... et voila.

PickleRick_0-1633804583365.png

So there must be something wrong with your extraction. Are you sure you didn't create the extractions for this sourcetype in wrong app and forgot to give permissions?

0 Karma

jnames10
Explorer

I am jealous! Can you share the source type settings or did you just accept the auto/defaults? any changes to line breaking or kv_mode? Thanks!

0 Karma

PickleRick
SplunkTrust
SplunkTrust

PickleRick_0-1634016177071.png

In fact all I did was add a KV_MODE entry. The rest was left as it was.

 

jnames10
Explorer

@PickleRickthanks for taking the time to do that. I fired up Splunk Enterprise and tried it and ok first time like you said. There is a complication around with my Splunk Cloud or just Splunk Cloud. Will do some more fiddling! Thanks again.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jnames10 

 

Is it OK if you have a search which can extract fields for you? If yes can you please share inn which fields you are interested?

KV

 

0 Karma

jnames10
Explorer

Hi Kamlesh,
Thanks for taking a look. I wondered if it might have to be search time extraction. The fields would be addr, name, portid, state

Ideally I want to end up with output I can go on to use in further searches, so if I had a table like below I could outputlookup it into a lookup table to use again.

addrnameportidstateportidstateportidstateand on for each portid value in event
81.123.123.123host.name.resolved.com21filtered22open23open 
         

 

or would I be better to have it like this?:

addrnameportidstate
81.123.123.123host.name.resolved.com21filtered
81.123.123.123host.name.resolved.com22open
81.123.123.123host.name.resolved.com23open
next ipnext nameetc etc 

 

 thanks again

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jnames10 

Can you please try this?

YOUR_SEARCH | spath
| rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0
| eval t=mvzip(portid,state) 
| mvexpand t
| eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1)
| table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state
| rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name

 

My Sample Search :

| makeresults 
| eval _raw="<host starttime=\"1633560153\" endtime=\"1633560291\"><status state=\"up\" reason=\"timestamp-reply\" reason_ttl=\"34\"/><address addr=\"81.123.123.123\" addrtype=\"ipv4\"/><hostnames><hostname name=\"host.name.resolved.com\" type=\"PTR\"/></hostnames><ports><port protocol=\"tcp\" portid=\"21\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ftp\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"22\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ssh\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"23\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"telnet\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"80\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"443\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"https\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8000\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-alt\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8080\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-proxy\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8888\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"sun-answerbook\" method=\"table\" conf=\"3\"/></port></ports><times srtt=\"15851\" rttvar=\"15851\" to=\"100000\"/></host>" 
| spath
| rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0
| eval t=mvzip(portid,state) 
| mvexpand t
| eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1)
| table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state
| rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name


 Output:

Screenshot 2021-10-10 at 10.48.56 AM.png

 

Thanks
KV
▄︻̷̿┻̿═━一   😉

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

jnames10
Explorer

@kamlesh_vaghelathank you very much for that. i was very helpful! I am able to build some tables. It looks like Splunk Cloud is treating the XML input differently and not importing it in the same way as Splunk Enterprise. I am still fiddling to try and get it working but the regex you've done is really appreciated!

Thank you!

0 Karma

RMcCurdyDOTcom
Explorer

 XtremeNmapParser ! to convert the xml to JSON and then used HEC to send it all to Spunk!

https://github.com/xtormin/XtremeNmapParser/issues/1

0 Karma

RMcCurdyDOTcom
Explorer

nmap XML to SPLUNK HEC !!! 

https://github.com/freeload101/SCRIPTS/blob/b3f83288a9f289d86f6cdd04898478d0427097ce/Bash/NMAP_FRUIT.sh#L80

 

0 Karma

RMcCurdyDOTcom
Explorer

 

got nasty gram for posting links

search online for freeload101 github in scripts nmap_fruit.sh

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...