Comparing events from 2 dates to detect new ones

[aagmon]

Hi All...

i'll first describe my scenario.. i have logs that contains entries regarding open ports like:

1-1-2000 192.168.0.1 port=80 service=http

1-1-2000 192.168.0.1 port=22 service=ssh

1-3-2000 192.168.0.1 port=80 service=http

1-3-2000 192.168.0.1 port=3350 service=unknown

1-3-2000 192.168.0.1 port=80 service=http

now' you can see that on the 1-3 an open port (3350) was detected, while that same port was not detected before.

how can i search for events like this? how can i compare results from scanA at a specific date to another one?

[gkanapathy · Accepted solution]

You can do:

sourcetype=myevents | eval when=if(_time>now()-600, "recent","older") | stats count, first(when) as mostrecent by port | where count < 2 AND mostrecent="recent"

and it will show things in the last 600 seconds that were not seen before that. This search will take a long time to run if you want to look back a long way to decide if you've seen a port "before".

HOWEVER, if this is something you have to run a lot (or schedule), it will be much more efficient to save the ports that have been seen in a lookup file as you go, which means you should schedule a search that runs, e.g., every 10 minutes:

sourcetype=myevents earliest=-20m latest=-10m | dedup port | fields port | eval seen="yes" | append [ inputlookup savedhosts.csv ] | dedup port | outputlookup savedhosts.csv

Then in your search to find events, do:

sourcetype=myevents earliest=-10min | lookup savedhosts.csv host OUTPUT seen | where NOT seen="yes"

to find events in the last 10 minutes that have not been saved to the lookup file. This search can be made more efficient by defining the lookup in props.conf:

[myevents]
LOOKUP-seen = savedhosts port OUTPUT seen

and transforms.conf

[savedhosts]
filename = savedhosts.csv

and search with:

sourcetype=myevents earliest=-10min NOT seen="yes"

[aagmon]

thanks.. that really is great.
just that in the log file i also have multiple hosts
e.g. it logs different hosts and ports:
do i need to use "dedup port,host" to solve it?