Splunk Search

Filtering windows security event logs with Regex

andrewdidone
Path Finder

Hi there.

We've been having issues with our DC's sending to much information across to Splunk and require assistance on creating some regex filtering strings, as we are not familiar with regex.

We are currently pulling windows security events from 2 Windows domain controllers and received issues with the amount events indexed which constantly violates or license.

We have windows logon events (event code: 4624) that capture both user information logons as well as machine logons. There are so many of these logon events that we dont need and would like to remove it in order to stay within the license limit.

The security events also have a large description included in the event under the event type "Message" that would like to be removed.

Here is an example of what we have:
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 2/28/2014 10:25:51 AM
Event ID: 4624
Task Category: Logon
Level: Information
Keywords: Audit Success
User: N/A
Computer: Computer
Description:
An account was successfully logged on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: DOMAIN*USERNAME*
Account Name: username
Account Domain: DOMAIN
Logon ID: 0xb008f014
Logon GUID: {877a24e2-7fff-857b-30a6-e4f061536b11}

Process Information:
Process ID: 0x0
Process Name: -

Network Information:
Workstation Name:

Source Network Address: IP address
Source Port: 49914

Detailed Authentication Information:
Logon Process: Kerberos
Authentication Package: Kerberos
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.


We recieve the same event for machine logons however it has the machine name with a $ in the name:

        Security ID:                   *DOMAIN*\*MACHINE*$
        Account Name:              *MACHINE*$

The request is pretty much this:

Create a regex for the props and transforms that will filter out ALL events that contain the "machine"$ and KEEP the events that contain a proper username. REMOVE the "Message" field from the events to reduce indexing size.

Any help will be greatly appreciated. Please let me know if it needs more clarification.

Thanks,
Andrew

0 Karma

splunk68
Path Finder

The best is to exclude the contents of a specific Event ID directly from the Universal Forwarder. Using a Heavy Forwarder for that is not necessary anymore.
Create an app to deploy and in the inputs.conf you can create multiple whitelist matching different regex.
So you can filter out specific messages or fields within a specific Event ID.

whitelist = | key=regex [key=regex]

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
,

0 Karma

hkust
New Member

Can you share the transforms.conf on removing the "Message" and "Body" in order to reduce indexing size? Thanks.

0 Karma

andrewdidone
Path Finder

The Heavy forwarder is just passing other information and is not indexing. It exists for other types of data sets. Apologies, I should have mentioned that.

0 Karma

lukejadamec
Super Champion

Place the props and transforms on the heavy forwarder.
You can only cook data once.

0 Karma

chanamoluk
Explorer

in the above log data , i want to filter all the text data(bottom 12 lines) through *inputs.con*f .

please help me out how to blacklist only the text data through inputs.conf

0 Karma

andrewdidone
Path Finder

6.0 on the Indexer and SearchHead, 5.0.5 on Heavy Forwarder.

0 Karma

lukejadamec
Super Champion

This should work for you:

props.conf

 [source::wineventlog:security]
 TRANSFORMS-drop = delFilter

transforms.conf

[delFilter]
REGEX = (?msi)^EventCode=4624\D.*Account\s+Name:\s+[a-z0-9-]+[\$]
DEST_KEY = queue
FORMAT = nullQueue

This will grab any account name made up of the characters a-z 0-9 and - that ends in a $. You should test this in a search regex first though because it is very possible that the machine name is as one of the account names in 100% of the 4624 events, which means you would be dropping all of them.

See this post:

http://answers.splunk.com/answers/102989/windows-event-filtering

lukejadamec
Super Champion

Try this in a search first:

EventCode=4624 | regex "(?msi)^EventCode=4624\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-z0-9-]+[\$]"

It should find all of the events you want to drop.

wrangler2x
Motivator

You can drop the i in msi if you add A-Z to the character class, viz:

EventCode=4624 | regex "(?ms)^EventCode=4624\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-zA-Z0-9-]+[\$]"

The i is for case insensitive, which is overhead you really don't need if you've accurately characterized the various strings (which you have).

0 Karma

mbrenton
Explorer

Thank you so much for this! I expanded on this idea and came up with the following two regex lines for transforms.conf (events 4624 and 4625 have two account name fields, 4634 and 4768-4771 only have one):

REGEX = (?msi)^EventCode=(4624|4625)\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-z0-9-]+[\$]

REGEX = (?msi)^EventCode=(4634|4768|4769|4770|4771)\D.*Account\s+Name:\s+[a-z0-9-]+[\$]

0 Karma

Nsdjanin
New Member

How would this regex look like if I'm interested in filtering out only one machine account - SERVER01$ and not all of them?

Please help!

0 Karma

leonsanm
New Member

thank you ; removed quotes! thank you

0 Karma

leonsanm
New Member

this works great in the search, but I tried this in my transforms and I still get the events... 😞

0 Karma

lukejadamec
Super Champion

Yes.... But it will take a few minutes.

0 Karma

andrewdidone
Path Finder

Hey, thanks for taking the time on this. Unfortunately i am still seeing the undesired events. Please see raw data below I believe whats causing an issue is the fact there are 2 Account Name fields in the event. Is there a way to do "Account Name: = "-" AND "machine$"

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: AUSODOM\0613-09015$
Account Name: 0613-09015$
Account Domain: AUSODOM
Logon ID: 0x157ad97b2
Logon GUID: {98EF322F-
Key Length: 0

Thanks,
Andrew

0 Karma

lukejadamec
Super Champion

Sorry for the delay. I updated the answer. Be careful that you are not dropping 100% of the 4624 events.

0 Karma

andrewdidone
Path Finder

Both, really. We don't want to see the events from the $ machines at all. Is that possible? I realize it's difficult with the Windows event logs format being the way they are.

0 Karma

lukejadamec
Super Champion

I see the problem. These are multi value fields, and the machine name with a $ can occur in 100% of them. Are you looking to drop the events where the $ is in the first or second Account_Name field?

0 Karma

andrewdidone
Path Finder

Ah ok. Yep they return as Security ID: and Account Name. I'll keep that in mind for the future.

0 Karma

lukejadamec
Super Champion

When using regex, base your regex on the return from -
EventCode=4624 |table _raw

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.