Failed Logon attempt Correlation

[pstephens93]

Hello,

Our environment has this linux server that continually get's hit with Brute force attacks. I am trying to figure out where they are coming from. Since our servers are behind a nated firewall I need to be able to see the failed logon attempts and match the server IP , port number  to the DestinationIP and natSourcePort

I am trying to use sub search. I have one search that searches all of our indexes for failed passwords from server IP and I want it to return the portnumber

Then do a second search that matches the IP and Port numbers. of the first search and return the top Source addresses 

Here is what I have so far. Any help would be apricated 

index="palo" ( PA_natDestinationIP=Server IP ANDPortNumber=PA_natSource_Port )
[ search index= "*" Failed password  IP = ServerIP | return PortNumber ]
| stats count by PA_SourceAddress |sort by count desc

[richgalloway]

What is your question?  You have a search, so what's the problem?  What help is needed?

You use "index=*" in a production query so no pudding for you tonight.  Take some time to figure out which indexes contain failed password events and have the query search only those indexes.  Your indexers will thank you for it.

[pstephens93]

So I have my two searches,
The inner search  finds the failed password attempts for a particular server ( I tested it and it works)  I need this to return a list of all the Port numbers used for the failed logon attempts . The return portion does not seem to work

The outer search will seach for Server IP(DestionationIP) and  take the list of port numbers returned from the first search and  match it with the nat port number.

I am having trouble with the return portion of the search s well as with the out search. I am not sure if the return function is not returning the list so it can not match up port numbers. 

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )
[ search index="wawf" (Failed password AND Linux_Server="server1") | return empherical_Port ]
| stats count by PA_SourceAddress |sort by count desc | Table  SourceAddress, Empherical_Port, NatSource_Port

[richgalloway]

The return command returns a single value by default.

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )
[ search index="wawf" (Failed password AND Linux_Server="server1") 
  | return 1000 empherical_Port ]
| stats count by PA_SourceAddress 
| sort by count desc 
| Table  SourceAddress, Empherical_Port, NatSource_Port

Run the subsearch by itself to see what it will return to the outer search. 

[pstephens93]

When I ran the sub Search on it's own I got a values like (ephemerical_Port="63450")  I added a $  to the fieldname and it dropped the Empherical_Port  and updated the  results to and example like (63450) 

I think the problem now is it is trying to look through and match up

Empherical_Port=natSource_Port

(63450) = 63450  which would show as not a match. 

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )
[ search index="wawf" (Failed password AND Linux_Server="server1") 
  | return 1000 $empherical_Port ]
| stats count by PA_SourceAddress 
| sort by count desc 
| Table  SourceAddress, Empherical_Port, NatSource_Port

[richgalloway]

What happens with subsearches is the results replace the subsearch in the outer search.  In our case, the search would become

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )
(63450)
| stats count by PA_SourceAddress 
| sort by count desc 
| Table  SourceAddress, Empherical_Port, NatSource_Port

This is valid, but inefficient (Splunk will look in all fields for "63450") and may return unexpected results (63450 may be something other than a port number).

The solution is to make sure the subsearch returns a field name known to the outer search.

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )
[ search index="wawf" (Failed password AND Linux_Server="server1") 
  | rename empherical_Port as Empherical_Port
  | return 1000 Empherical_Port ]
| stats count by PA_SourceAddress 
| sort by count desc 
| Table  SourceAddress, Empherical_Port, NatSource_Port

Please confirm Empherical_Port and natSource_Port are both present in the events.  If they are not both present then the search will find nothing.

[pstephens93]

I am still having issues with this. The sub search returns the following:

When I run the full search together I get nothing, When I just run the outer search with hard coded values  
   I get a match. When I do Empherical_Port = 20961 which is in  the port values list about 200 values into the list.  When the searches are combined I get nothing, when I do the inner search then hard code the values into the outer search. Some get hits some do not.  Is it not running through the whole return list? 

index="palo" ( natDestinationIP=10.63.16.103 AND Empherical_Port=natSource_Port )

index="palo" ( natDestinationIP = 10.63.16.103 AND natSourcePort =20961 )

[richgalloway]

The subsearch is returning a field called "Port".  Does the outer search have such a field?  I suspect it does not since you use Empherical_Port in your example.  The subsearch must return a field name known to the outer search to produce results.

[pstephens93]

The Inner search I renamed the Field ephemeral_Port to Port.

When I break down the Searches  into the following:

 index="wawf-prod" (Failed password AND Linux_Server="wawfprodftpd37") 
|rename ephemeral_Port as Port
|return 1000 Port

 This Returns the list of Ports as shown in the above screen shot I sent.  

When I run the Code below with a hard coded value for the  PA_natSourcePort = 20961 It returns a match. the port number 20961 is one of the values returned from the first search that I hard coded to see if It was a values issue. Which some of the values hard code match and some do not. 

index="palo-test"  PA_natDestinationIP=10.63.16.103 AND PA_natSourcePort = 20961

So Something is happening in the PA_natSourcePort = port  when the two searches are combined with a sub search. Any ideas why that is? 

index="palo-test"  ( PA_natDestinationIP=10.63.16.103 AND PA_natSourcePort = Port)
[ search index="wawf-prod" (Failed password AND Linux_Server="wawfprodftpd37") 
|rename ephemeral_Port as Port
|return 1000 Port ] 
| stats count by PA_SourceAddress |sort by count desc | table PA_SourceAddress, Port, PA_natSourcePort, count

[richgalloway · Accepted solution]

At the risk of repeating myself, the subsearch must return field names that appear in index=wawf-prod.  If it doesn't then the main search will find no results.