Filtering windows security event logs with Regex

[andrewdidone]

Hi there.

We've been having issues with our DC's sending to much information across to Splunk and require assistance on creating some regex filtering strings, as we are not familiar with regex.

We are currently pulling windows security events from 2 Windows domain controllers and received issues with the amount events indexed which constantly violates or license.

We have windows logon events (event code: 4624) that capture both user information logons as well as machine logons. There are so many of these logon events that we dont need and would like to remove it in order to stay within the license limit.

The security events also have a large description included in the event under the event type "Message" that would like to be removed.

Here is an example of what we have:
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 2/28/2014 10:25:51 AM
Event ID: 4624
Task Category: Logon
Level: Information
Keywords: Audit Success
User: N/A
Computer: Computer
Description:
An account was successfully logged on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: DOMAIN*USERNAME*
Account Name: username
Account Domain: DOMAIN
Logon ID: 0xb008f014
Logon GUID: {877a24e2-7fff-857b-30a6-e4f061536b11}

Process Information:
Process ID: 0x0
Process Name: -

Network Information:
Workstation Name:

Source Network Address: IP address
Source Port: 49914

Detailed Authentication Information:
Logon Process: Kerberos
Authentication Package: Kerberos
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

---

We recieve the same event for machine logons however it has the machine name with a $ in the name:

        Security ID:                   *DOMAIN*\*MACHINE*$
        Account Name:              *MACHINE*$

The request is pretty much this:

Create a regex for the props and transforms that will filter out ALL events that contain the "machine"$ and KEEP the events that contain a proper username. REMOVE the "Message" field from the events to reduce indexing size.

Any help will be greatly appreciated. Please let me know if it needs more clarification.

Thanks,
Andrew

[splunk68]

The best is to exclude the contents of a specific Event ID directly from the Universal Forwarder. Using a Heavy Forwarder for that is not necessary anymore.
Create an app to deploy and in the inputs.conf you can create multiple whitelist matching different regex.
So you can filter out specific messages or fields within a specific Event ID.

whitelist = | key=regex [key=regex]

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
,

[hkust]

Can you share the transforms.conf on removing the "Message" and "Body" in order to reduce indexing size? Thanks.

[andrewdidone]

The Heavy forwarder is just passing other information and is not indexing. It exists for other types of data sets. Apologies, I should have mentioned that.

[lukejadamec]

Place the props and transforms on the heavy forwarder.
You can only cook data once.

[chanamoluk]

in the above log data , i want to filter all the text data(bottom 12 lines) through *inputs.con*f .

please help me out how to blacklist only the text data through inputs.conf

[andrewdidone]

6.0 on the Indexer and SearchHead, 5.0.5 on Heavy Forwarder.

[lukejadamec]

This should work for you:

props.conf

 [source::wineventlog:security]
 TRANSFORMS-drop = delFilter

transforms.conf

[delFilter]
REGEX = (?msi)^EventCode=4624\D.*Account\s+Name:\s+[a-z0-9-]+[\$]
DEST_KEY = queue
FORMAT = nullQueue

This will grab any account name made up of the characters a-z 0-9 and - that ends in a $. You should test this in a search regex first though because it is very possible that the machine name is as one of the account names in 100% of the 4624 events, which means you would be dropping all of them.

See this post:

http://answers.splunk.com/answers/102989/windows-event-filtering

[lukejadamec]

Try this in a search first:

EventCode=4624 | regex "(?msi)^EventCode=4624\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-z0-9-]+[\$]"

It should find all of the events you want to drop.

[wrangler2x]

You can drop the i in msi if you add A-Z to the character class, viz:

EventCode=4624 | regex "(?ms)^EventCode=4624\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-zA-Z0-9-]+[\$]"

The i is for case insensitive, which is overhead you really don't need if you've accurately characterized the various strings (which you have).

[mbrenton]

Thank you so much for this! I expanded on this idea and came up with the following two regex lines for transforms.conf (events 4624 and 4625 have two account name fields, 4634 and 4768-4771 only have one):

REGEX = (?msi)^EventCode=(4624|4625)\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-z0-9-]+[\$]

REGEX = (?msi)^EventCode=(4634|4768|4769|4770|4771)\D.*Account\s+Name:\s+[a-z0-9-]+[\$]

[Nsdjanin]

How would this regex look like if I'm interested in filtering out only one machine account - SERVER01$ and not all of them?

Please help!

[leonsanm]

thank you ; removed quotes! thank you

[leonsanm]

this works great in the search, but I tried this in my transforms and I still get the events... 😞

[lukejadamec]

Yes.... But it will take a few minutes.

[andrewdidone]

Hey, thanks for taking the time on this. Unfortunately i am still seeing the undesired events. Please see raw data below I believe whats causing an issue is the fact there are 2 Account Name fields in the event. Is there a way to do "Account Name: = "-" AND "machine$"

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: AUSODOM\0613-09015$
Account Name: 0613-09015$
Account Domain: AUSODOM
Logon ID: 0x157ad97b2
Logon GUID: {98EF322F-
Key Length: 0

Thanks,
Andrew

[lukejadamec]

Sorry for the delay. I updated the answer. Be careful that you are not dropping 100% of the 4624 events.

[andrewdidone]

Both, really. We don't want to see the events from the $ machines at all. Is that possible? I realize it's difficult with the Windows event logs format being the way they are.

[lukejadamec]

I see the problem. These are multi value fields, and the machine name with a $ can occur in 100% of them. Are you looking to drop the events where the $ is in the first or second Account_Name field?

[andrewdidone]

Ah ok. Yep they return as Security ID: and Account Name. I'll keep that in mind for the future.

[lukejadamec]

When using regex, base your regex on the return from -
EventCode=4624 |table _raw