Join 2 events with same "source"

[henryyiu2degree]

I want to join the nmap scanning results. The common field is the source "nmapscan_1.gnmap" while other scans will have a different source name.

Event 1

# Nmap 5.51 scan initiated Tue Dec 11 10:54:16 2017 as: nmap -A -T4 -oA scan_192.168.1.0_24_20171219 192.168.1.0/24
host =nmapserver   source =nmapscan_1.gnmap   sourcetype =nmap

Event 2

# Nmap done at Tue Dec 11 12:20:04 2017 -- 256 IP addresses (81 hosts up) scanned in 5147.70 seconds
host =nmapserver   source =nmapscan_1.gnmap   sourcetype =nmap

I want to get the following results:

Time, Subnet, #Hosts up, Duration
Tue Dec 11 10:54:16 2017, 192.168.1.0/24, 81, 5147.70

Could I use the "source" as a common field and how to do this? Thanks.

[micahkemp · Accepted solution]

If each file monitored represents a different scan then you can definitely use source for transaction.

Try this:

| rex "scan_(?<network>([0-9]{1,3}\.){3}[0-9]{1,3})_(?<masklen>[0-9]+)_"
| eval subnet=network."/".masklen
| rex "\((?<hosts_up>[0-9]+) hosts up\)"
| transaction startswith=eval(isnotnull(subnet)) endswith=eval(isnotnull(hosts_up)) source
| table _time subnet hosts_up duration

[henryyiu2degree]

This works perfectly, thanks for the help.

I have changed a little to use the last subnet as that one is generated by nmap:

| rex field=_raw ".*scan\sinitiated\s.*\s(?<subnet>\d+.\d+.\d+.\d+\/\d\d)"
| rex field=_raw "\((?<hosts_up>\d+)\shosts\sup\)\sscanned\sin\s(?<duration>\d+).\d\d\sseconds"
| transaction startswith=eval(isnotnull(subnet)) endswith=eval(isnotnull(hosts_up)) source
| table _time, subnet, hosts_up, duration

[mayurr98]

hey @henryyiu2degrees

No, you cannot use the source as a common field. As in that particular source field, there might be plenty of events so essentially there is nothing common as such in these particular two events. Also, the source is metadata field it is not something that is coming from logs.
So if you want to correlate these two events, then you need to have something common in the raw logs i.e. host_ip OR username userid which is a unique identifier only for those particular two events.

I hope this explanation helps you understand these things!

[mayurr98]

Also try this

index=index1  OR index=index2  (source=source1 OR source=source2)
| rex field=_raw "gtid\((?P<trans_id>[^\)]+)"
| stats values(_raw) as raw_event by trans_id

Let me know what you get

[henryyiu2degree]

Thanks @mayurr98 for your comment.

I afraid there is no common fields in the raw logs but could I join "scan initiated" and the next "Nmap done" together?

[mayurr98]

I don’t think but you can try below search

index=your_index | transaction startswith=“scan initiated” endswith=“Nmap done”

After running this search see if like events are getting correlated although it is not recommended to use this command unless you have common field but just check !

Let me know if this works!

[henryyiu2degree]

I have tried this and work as well, but the others answer is more accurate.
Appreciate your help, thanks a lot.