How to identify an elusive process? (Help with investigating an IOC)

[eliasit]

Hello Splunkers,
I'm hoping to get some help investigating a potential IOC.

Here is the situation, my Infosec app for Splunk (https://splunkbase.splunk.com/app/4240/) dashboard has flagged a system for possible bruteforce attempt. I have identified the system (a workstation), the account name being attempted ("host"), the logon target (Domain) and some what of a pattern for attempts (2-3 attempts per 30 minute block and decreasing port number with each attempt for as long as the system is on regardless of whether or not a user is logged in).

What I can't find out, is the process involved. (I can't figure out what is trying to login) I tried google searching for malware that has this same behavior but didn't find anything. I have used "TcpLogView" in an attempt to log the process that is opening the port but, it seems that it opens and closes too quickly for TcpLogView to see it, as the ports listed in the events are not listed by TcpLogView. I have even installed Splunk forwarder on this workstation to gain some visibility but I don't know if I'm ingesting the right data.

So, how can I identify what is making the attempts?
Thank you in advance for any help or suggestions.

Update: These login attempts continue even if the system is running in safemode (with networking).

Here is a log entry showing the failed attempt with ##comments and some *redaction. This event was generated by the DC.
09/10/2019 02:31:04 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4768
EventType=0
Type=Information
ComputerName=*DC-Name
TaskCategory=Kerberos Authentication Service
OpCode=Info
RecordNumber=3587264684
Keywords=Audit Failure
Message=A Kerberos authentication ticket (TGT) was requested.

Account Information:
Account Name: host
Supplied Realm Name: *Domain-name
User ID: NULL SID

Service Information:
Service Name: krbtgt/*Domain-name
Service ID: NULL SID

Network Information:
Client Address: 192.168.10.170 ##workstation's IP
Client Port: 53169

Additional Information:
Ticket Options: 0x40800010
Result Code: 0x6 ##Client not found in Kerberos database
Ticket Encryption Type: 0xFFFFFFFF
Pre-Authentication Type: -

[eliasit]

@zacharychristensen @igifrin_splunk

Hello again splunkers,
Still stuck on this one.
So I have sysmon data coming in and can see processes making network connections in the sysmon events but there are no sysmon events for the ports listed in the Windows security log entries for the same time periods.

For example, here is the search I used with the time picker set to last 60 minutes.

tag=authentication  Keywords="Audit Failure"  Account_Name=host src=192.168.10.170

This returns security log events as shown in the original post with different port numbers. (53475, 53476, 53477, 53478, 53479, 53480, 53481, 53482, 56153, 56154)

But when I search the sysmon data for those ports during the same time frame I get no results.

source="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=3 SourceIp=192.168.10.170 SourcePort=53480

Am I doing something wrong? Or do I need to ingest other data?

Thanks for any help you can offer.

[zacharychristen]

Are you ingesting logs from the source IP that is causing the failures? Or are you just logging the events on the DC?

Also, are these attempts occurring outside of normal business hours? How frequent are they occurring? These types of failures could often indicate some type of misconfiguration that needs to be updated. It is difficult to tell without knowing all the contextual information.

I would always error on saying, yes, provide more visibility through other data sources, but that comes at a cost.

I would probably start on the device that is causing the failures (I think this would be the workstation sitting at 192.168.10.170 from your example), is there any other anomalous activity occurring with this device? Have you scanned the device with an AV scanner? Does the device also have many local login failures? (event 4625 I believe). Do you have endpoint protection software installed on the device? Are there other devices that are generating a similar error?

Visibility is key, and if you don't currently have the ability to monitor that device directly, it may be worth it to investigate the machine manually. If there is higher suspicion that this is an actual offense, pull it from the network and perform scans on the device.

[eliasit]

@zacharychristensen
Thanks for the response.

Are you ingesting logs from the source
IP that is causing the failures?

Yes, both windows security logs and sysmon logs. (from several workstations)

Also, are these attempts occurring
outside of normal business hours?
How frequent are they occurring?

They happen whenever the system is on, whether a user is logged in or not and about every 30 minutes. Also, they happen even if the system is booted in safemode (with networking).

I would always error on saying, yes,
provide more visibility through other
data sources,...

What would you recommend?

Is there any other anomalous activity
occurring with this device?

Not that we have noticed.

Have you scanned the device with an AV
scanner?
Yes, it comes back clean and the AV software also scans on access.
Does the device also have many local
login failures? (event 4625 I
believe).

No. And you are correct, EventID 4625.

Do you have endpoint protection
software installed on the device?

No.

Are there other devices that are
generating a similar error?

Yes. 2 other workstations. Of the 3 total workstations, 2 are used regularly (at the employee's desk) and one is used in a conference room but most of the time it is off unless I'm doing Windows updates to it.

Also, I don't know if it matters but this domain was originally a Windows 2003 domain that was not raised to 2008 domain level until 2018 and then 2012 domain level in 2019. The domain has never been recreated from scratch. (I.E. Potential misconfigurations could have been piling up for years.)

Thanks again

Edit: fixed formatting

[zacharychristen · Accepted solution]

So this really depends on what type of logs you are ingesting into Splunk. If you have endpoint protection logs, such as crowdstrike or carbon black, you can use these to correlate processes being run on the machine. If sysmon is being utilized, these logs can also be very helpful. If the only logs you are ingesting from the device are standard windows logs, then this can be a bit tricky as Splunk is only aware of the data being sent to it (which in this case doesn't look to be processes running on the device). In that case, you may have to manually investigate the device. Autoruns64 from sysinternals is a handy tool to use as most malware will show on startup for persistence.

[eliasit]

@zacharychristensen @igifrin_splunk Thank you both for your responses. I was not ingesting sysmon data before so I have started the DSOG for that. I hope to have the bugs worked by next week.
Thanks again

[igifrin_splunk]

Just to add to the great points @zacharychristensen made - the InfoSec app has basic endpoint data analytics capabilities (in the latest version 1.4 under Search > Experimental Dashboards > Endpoints). Data from endpoint solutions like Sysmon, osquery, CarbonBlack, CrowdStrike will show up there as long as the data is onboarded in a CIM-compliant manner (typically with a CIM-compliant add-on from Splunkbase).