Splunk Search

Comparing events from 2 dates to detect new ones

aagmon
New Member

Hi All...

i'll first describe my scenario.. i have logs that contains entries regarding open ports like:

1-1-2000 192.168.0.1 port=80 service=http

1-1-2000 192.168.0.1 port=22 service=ssh

1-3-2000 192.168.0.1 port=80 service=http

1-3-2000 192.168.0.1 port=3350 service=unknown

1-3-2000 192.168.0.1 port=80 service=http

now' you can see that on the 1-3 an open port (3350) was detected, while that same port was not detected before.

how can i search for events like this? how can i compare results from scanA at a specific date to another one?

Tags (1)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You can do:

sourcetype=myevents | eval when=if(_time>now()-600, "recent","older") | stats count, first(when) as mostrecent by port | where count < 2 AND mostrecent="recent"

and it will show things in the last 600 seconds that were not seen before that. This search will take a long time to run if you want to look back a long way to decide if you've seen a port "before".

HOWEVER, if this is something you have to run a lot (or schedule), it will be much more efficient to save the ports that have been seen in a lookup file as you go, which means you should schedule a search that runs, e.g., every 10 minutes:

sourcetype=myevents earliest=-20m latest=-10m | dedup port | fields port | eval seen="yes" | append [ inputlookup savedhosts.csv ] | dedup port | outputlookup savedhosts.csv

Then in your search to find events, do:

sourcetype=myevents earliest=-10min | lookup savedhosts.csv host OUTPUT seen | where NOT seen="yes"

to find events in the last 10 minutes that have not been saved to the lookup file. This search can be made more efficient by defining the lookup in props.conf:

[myevents]
LOOKUP-seen = savedhosts port OUTPUT seen

and transforms.conf

[savedhosts]
filename = savedhosts.csv

and search with:

sourcetype=myevents earliest=-10min NOT seen="yes"

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You can do:

sourcetype=myevents | eval when=if(_time>now()-600, "recent","older") | stats count, first(when) as mostrecent by port | where count < 2 AND mostrecent="recent"

and it will show things in the last 600 seconds that were not seen before that. This search will take a long time to run if you want to look back a long way to decide if you've seen a port "before".

HOWEVER, if this is something you have to run a lot (or schedule), it will be much more efficient to save the ports that have been seen in a lookup file as you go, which means you should schedule a search that runs, e.g., every 10 minutes:

sourcetype=myevents earliest=-20m latest=-10m | dedup port | fields port | eval seen="yes" | append [ inputlookup savedhosts.csv ] | dedup port | outputlookup savedhosts.csv

Then in your search to find events, do:

sourcetype=myevents earliest=-10min | lookup savedhosts.csv host OUTPUT seen | where NOT seen="yes"

to find events in the last 10 minutes that have not been saved to the lookup file. This search can be made more efficient by defining the lookup in props.conf:

[myevents]
LOOKUP-seen = savedhosts port OUTPUT seen

and transforms.conf

[savedhosts]
filename = savedhosts.csv

and search with:

sourcetype=myevents earliest=-10min NOT seen="yes"

aagmon
New Member

thanks.. that really is great.
just that in the log file i also have multiple hosts
e.g. it logs different hosts and ports:
do i need to use "dedup port,host" to solve it?

0 Karma
Get Updates on the Splunk Community!

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL

Splunk AI Assistant for SPL | Key Use Cases to Unlock the Power of SPL  The Splunk AI Assistant for SPL ...

Buttercup Games: Further Dashboarding Techniques (Part 5)

This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the ...

Customers Increasingly Choose Splunk for Observability

For the second year in a row, Splunk was recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for ...