I have been trying to get nmap output into Splunk. I thought the xml output would be nice and straightforward!
Whilst the events are separated, the issue is having multiple values of the same field in an event.
automatic field extractions pick up the first <port... section but the rest are ignored. I tried using KV_MODE=xml but that didn't make a difference. I thought Splunk was quite happy pulling in multi values with xml but maybe its not quite the xml Splunk is expecting.
I am on Splunk Cloud so cli changes are not an option.
Any pointers appreciated! Thanks.
<host starttime="1633560153" endtime="1633560291"><status state="up" reason="timestamp-reply" reason_ttl="34"/>
<address addr="81.123.123.123" addrtype="ipv4"/>
<hostnames>
<hostname name="host.name.resolved.com" type="PTR"/>
</hostnames>
<ports><port protocol="tcp" portid="21"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="ftp" method="table" conf="3"/></port>
<port protocol="tcp" portid="22"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="ssh" method="table" conf="3"/></port>
<port protocol="tcp" portid="23"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="telnet" method="table" conf="3"/></port>
<port protocol="tcp" portid="80"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http" method="table" conf="3"/></port>
<port protocol="tcp" portid="443"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="https" method="table" conf="3"/></port>
<port protocol="tcp" portid="8000"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http-alt" method="table" conf="3"/></port>
<port protocol="tcp" portid="8080"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="http-proxy" method="table" conf="3"/></port>
<port protocol="tcp" portid="8888"><state state="filtered" reason="no-response" reason_ttl="0"/><service name="sun-answerbook" method="table" conf="3"/></port>
</ports>
<times srtt="15851" rttvar="15851" to="100000"/>
</host>
Tried to replicate your problem. Created a simple test xml sourcetype, imported the event... et voila.
So there must be something wrong with your extraction. Are you sure you didn't create the extractions for this sourcetype in wrong app and forgot to give permissions?
I am jealous! Can you share the source type settings or did you just accept the auto/defaults? any changes to line breaking or kv_mode? Thanks!
In fact all I did was add a KV_MODE entry. The rest was left as it was.
@PickleRickthanks for taking the time to do that. I fired up Splunk Enterprise and tried it and ok first time like you said. There is a complication around with my Splunk Cloud or just Splunk Cloud. Will do some more fiddling! Thanks again.
Is it OK if you have a search which can extract fields for you? If yes can you please share inn which fields you are interested?
KV
Hi Kamlesh,
Thanks for taking a look. I wondered if it might have to be search time extraction. The fields would be addr, name, portid, state
Ideally I want to end up with output I can go on to use in further searches, so if I had a table like below I could outputlookup it into a lookup table to use again.
addr | name | portid | state | portid | state | portid | state | and on for each portid value in event |
81.123.123.123 | host.name.resolved.com | 21 | filtered | 22 | open | 23 | open | |
or would I be better to have it like this?:
addr | name | portid | state |
81.123.123.123 | host.name.resolved.com | 21 | filtered |
81.123.123.123 | host.name.resolved.com | 22 | open |
81.123.123.123 | host.name.resolved.com | 23 | open |
next ip | next name | etc etc |
thanks again
Can you please try this?
YOUR_SEARCH | spath
| rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0
| eval t=mvzip(portid,state)
| mvexpand t
| eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1)
| table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state
| rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name
My Sample Search :
| makeresults
| eval _raw="<host starttime=\"1633560153\" endtime=\"1633560291\"><status state=\"up\" reason=\"timestamp-reply\" reason_ttl=\"34\"/><address addr=\"81.123.123.123\" addrtype=\"ipv4\"/><hostnames><hostname name=\"host.name.resolved.com\" type=\"PTR\"/></hostnames><ports><port protocol=\"tcp\" portid=\"21\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ftp\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"22\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"ssh\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"23\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"telnet\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"80\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"443\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"https\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8000\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-alt\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8080\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"http-proxy\" method=\"table\" conf=\"3\"/></port><port protocol=\"tcp\" portid=\"8888\"><state state=\"filtered\" reason=\"no-response\" reason_ttl=\"0\"/><service name=\"sun-answerbook\" method=\"table\" conf=\"3\"/></port></ports><times srtt=\"15851\" rttvar=\"15851\" to=\"100000\"/></host>"
| spath
| rex field=_raw "portid=\"(?<portid>\d+)\"\>\<state\sstate=\"(?<state>[^\"]+)\"" max_match=0
| eval t=mvzip(portid,state)
| mvexpand t
| eval portid=mvindex(split(t,","),0), state=mvindex(split(t,","),1)
| table "host.address{@addr}" "host.hostnames.hostname{@name}" portid state
| rename "host.address{@addr}" as addr, "host.hostnames.hostname{@name}" as name
Output:
Thanks
KV
▄︻̷̿┻̿═━一 😉
If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.
@kamlesh_vaghelathank you very much for that. i was very helpful! I am able to build some tables. It looks like Splunk Cloud is treating the XML input differently and not importing it in the same way as Splunk Enterprise. I am still fiddling to try and get it working but the regex you've done is really appreciated!
Thank you!
XtremeNmapParser ! to convert the xml to JSON and then used HEC to send it all to Spunk!
https://github.com/xtormin/XtremeNmapParser/issues/1
nmap XML to SPLUNK HEC !!!
https://github.com/freeload101/SCRIPTS/blob/b3f83288a9f289d86f6cdd04898478d0427097ce/Bash/NMAP_FRUIT.sh#L80
got nasty gram for posting links
search online for freeload101 github in scripts nmap_fruit.sh