topic Re: How to improve the performance of my search to find new opened ports in All Apps and Add-ons

How to improve the performance of my search to find new opened ports

wweiland — Thu, 09 Feb 2017 17:21:35 GMT

Hi,

I'm trying to find new ports that are opened up on a system where I have 24 hours of existing data.

sourcetype=Unix:ListeningPorts | join host [search sourcetype=Unix:ListeningPorts | bucket span=1h _time | dedup _time,host | stats count AS hostcount by host | table host,hostcount] | bucket span=1h _time | dedup host,dest_port,_time | stats values(hostcount) AS hostcount count as num_data_samples by host,dest_port | where hostcount = 24 AND num_data_samples = 1

The search takes few minutes to complete and i'm trying to get the time down to run as an alert. I believe the subsearch is causing the delay, but i'm not sure how to get the number of times a host reported in a 24 hour period. If I don't use the subsearch then I can't do the dedup to remove the multiple entries per hour (multiple dest_port) for a host.

Does anyone have any suggestions on how to make this better?

TIA

Re: How to improve the performance of my search to find new opened ports

somesoni2 — Thu, 09 Feb 2017 17:26:37 GMT

Give this a try

sourcetype=Unix:ListeningPorts | bucket span=1h _time | dedup host,dest_port,_time | stats dc(_time) AS hostcount count as num_data_samples by host,dest_port | where hostcount = 24 AND num_data_samples = 1

Re: How to improve the performance of my search to find new opened ports

wweiland — Thu, 09 Feb 2017 18:47:36 GMT

so simple!!

Re: How to improve the performance of my search to find new opened ports

DalJeanis — Thu, 09 Feb 2017 20:45:28 GMT

Yep, same conclusion I came to.

The inner query was not calculating a hostcount, but an hourcount for how many hours a host was active over the time period.

However, I'm not sure what the business meaning of "I have 24 hours of data " is.

Re: How to improve the performance of my search to find new opened ports

tred23 — Thu, 09 Feb 2017 20:54:00 GMT

This is an awesome idea!

Re: How to improve the performance of my search to find new opened ports

wweiland — Thu, 09 Feb 2017 21:00:05 GMT

I wanted to avoid false positives for systems being turned up in an active environment.

Re: How to improve the performance of my search to find new opened ports

DalJeanis — Thu, 09 Feb 2017 21:05:40 GMT

Do you mean, "Which have been running every hour for the past 24 hours"? or "Which I have been monitoring for at least 24 hours?"

sourcetype=Unix:ListeningPorts 
| table  _time, host, dest_port 
| bin _time span=1h 
| eventstats dc(_time) as OpHours, min(_time) as FirstHostEvent by host
| stats min(_time) as FirstPortEvent, max(OpHours) as OpHours, max(FirstHostEvent) as FirstHostEvent by host, dest_port
| table host, FirstHostEvent, OpHours, dest_port, FirstPortEvent
| sort 0 host dest_port
| where FirstHostEvent < relative_time(now(),"-1d") AND  FirstPortEvent > relative_time(now(),"-1h")

The last comparison assuming an hourly scan.

Re: How to improve the performance of my search to find new opened ports

wweiland — Thu, 09 Feb 2017 21:08:31 GMT

Have been monitored for at least 24 hours.

Re: How to improve the performance of my search to find new opened ports

somesoni2 — Thu, 09 Feb 2017 21:08:46 GMT

I believe you missed field OpHours and FirstHostEvent to be included in stats.

Re: How to improve the performance of my search to find new opened ports

DalJeanis — Thu, 09 Feb 2017 21:11:57 GMT

crud, you're right.

Hey, I added them using eventstats, so they're scrolled off to the right in my head from all the other fields. 😉

Re: How to improve the performance of my search to find new opened ports

DalJeanis — Thu, 09 Feb 2017 21:17:40 GMT

This strategy is going to throw an alert whenever the last use of a port scrolls off the time horizon that you are searching. Might want to merge in an inputcsv and send out an outputcsv with the known hosts and ports.

Re: How to improve the performance of my search to find new opened ports

wweiland — Thu, 09 Feb 2017 21:19:37 GMT

Ugh, good point. I'll have to think that through.

Thanks again!!!

Re: How to improve the performance of my search to find new opened ports

wweiland — Tue, 29 Sep 2020 12:49:45 GMT

I believe this approach would account for systems where applications are being decommissioned. Here i'm only looking for ports = 1 and the last recorded time was within the last hour from the end of the search. Does this look right?

|pivot SecOps__Listening_Ports Unix_Listening_Ports SPLITROW _time, SPLITROW host, SPLITROW dest_port | eventstats last(_time) AS maxtime | bucket span=1h _time | dedup host,dest_port,_time,maxtime | stats values(maxtime) AS maxtime last(_time) AS lasttime dc(_time) AS hostcount count as portcount by host,dest_port | where hostcount >= 24 AND portcount = 1 AND lasttime >= relative_time(maxtime, "-1h")

Sorry, also switched it to a pivot table.