topic Re: Filtering windows security event logs with Regex in Splunk Search

Filtering windows security event logs with Regex

andrewdidone — Fri, 28 Feb 2014 02:17:47 GMT

Hi there.

We've been having issues with our DC's sending to much information across to Splunk and require assistance on creating some regex filtering strings, as we are not familiar with regex.

We are currently pulling windows security events from 2 Windows domain controllers and received issues with the amount events indexed which constantly violates or license.

We have windows logon events (event code: 4624) that capture both user information logons as well as machine logons. There are so many of these logon events that we dont need and would like to remove it in order to stay within the license limit.

The security events also have a large description included in the event under the event type "Message" that would like to be removed.

Here is an example of what we have:
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 2/28/2014 10:25:51 AM
Event ID: 4624
Task Category: Logon
Level: Information
Keywords: Audit Success
User: N/A
Computer: Computer
Description:
An account was successfully logged on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: DOMAIN*USERNAME*
Account Name: username
Account Domain: DOMAIN
Logon ID: 0xb008f014
Logon GUID: {877a24e2-7fff-857b-30a6-e4f061536b11}

Process Information:
Process ID: 0x0
Process Name: -

Network Information:
Workstation Name:

Source Network Address: IP address
Source Port: 49914

Detailed Authentication Information:
Logon Process: Kerberos
Authentication Package: Kerberos
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

We recieve the same event for machine logons however it has the machine name with a $ in the name:

        Security ID:                   *DOMAIN*\*MACHINE*$
        Account Name:              *MACHINE*$

The request is pretty much this:

Create a regex for the props and transforms that will filter out ALL events that contain the "machine"$ and KEEP the events that contain a proper username. REMOVE the "Message" field from the events to reduce indexing size.

Any help will be greatly appreciated. Please let me know if it needs more clarification.

Thanks,
Andrew

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 02:24:04 GMT

What version of Splunk?

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 02:37:50 GMT

This should work for you:

props.conf

 [source::wineventlog:security]
 TRANSFORMS-drop = delFilter

transforms.conf

[delFilter]
REGEX = (?msi)^EventCode=4624\D.*Account\s+Name:\s+[a-z0-9-]+[\$]
DEST_KEY = queue
FORMAT = nullQueue

This will grab any account name made up of the characters a-z 0-9 and - that ends in a $. You should test this in a search regex first though because it is very possible that the machine name is as one of the account names in 100% of the 4624 events, which means you would be dropping all of them.

See this post:

http://answers.splunk.com/answers/102989/windows-event-filtering

Re: Filtering windows security event logs with Regex

andrewdidone — Fri, 28 Feb 2014 03:07:01 GMT

Hi lukejadamec

Thanks for the quick reply. I tried this and sadly it didn't seem to work.

Breakdown of what I did:
-Created and placed props.conf and transforms.conf with config supplied above into the system/local folder of the Indexer.
-Restarted splunk via command line: "splunk restart".

I changed the time picker to 1 minute ago. Unfortunately I am still seeing the "Message" field as well as the events from the "machine"$ hots.
Would you have any further ideas on this. Anything will greatly appreciated.

Cheers,
Andrew

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 03:22:38 GMT

I can test it in the morning.
Otherwise, the way you set it up looks good except - this does not target the Message field. You should be able to test the regex in a search string to fix it:
index=main EventCode=4624 | regex "(?msi)^EventCode=4624\D.*^Security\sID:\s+[^ ]\$.*^Account\s+Name:\s+[^ ]\$" |table EventCode
That should pull only 4624 events as specified, if you get nothing, then the regex needs fixing.
You can start small:
index=main EventCode=4624 | regex "(?msi)^EventCode=4624\D"
And grow:
index=main EventCode=4624 | regex "(?msi)^EventCode=4624\D.*^Security\sID:\s+[^ ]\$"

Re: Filtering windows security event logs with Regex

andrewdidone — Fri, 28 Feb 2014 03:33:08 GMT

6.0 on the Indexer and SearchHead, 5.0.5 on Heavy Forwarder.

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 03:36:27 GMT

Place the props and transforms on the heavy forwarder.
You can only cook data once.

Re: Filtering windows security event logs with Regex

andrewdidone — Mon, 28 Sep 2020 16:00:44 GMT

Thanks for that.
The small string returns events, but the "grow" string and initial search string does not.
Also, I should mention that the "Account Name" and "Seurity ID" are labeled as "Account_Name" and "Security_ID" in the Splunk fields. Perhaps that causes issues int the regex?

Re: Filtering windows security event logs with Regex

andrewdidone — Fri, 28 Feb 2014 03:52:20 GMT

The Heavy forwarder is just passing other information and is not indexing. It exists for other types of data sets. Apologies, I should have mentioned that.

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 03:53:12 GMT

When using regex, base your regex on the return from -
EventCode=4624 |table _raw

Re: Filtering windows security event logs with Regex

andrewdidone — Fri, 28 Feb 2014 04:04:27 GMT

Ah ok. Yep they return as Security ID: and Account Name. I'll keep that in mind for the future.

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 28 Feb 2014 15:21:01 GMT

I see the problem. These are multi value fields, and the machine name with a $ can occur in 100% of them. Are you looking to drop the events where the $ is in the first or second Account_Name field?

Re: Filtering windows security event logs with Regex

andrewdidone — Sun, 02 Mar 2014 22:37:27 GMT

Both, really. We don't want to see the events from the $ machines at all. Is that possible? I realize it's difficult with the Windows event logs format being the way they are.

Re: Filtering windows security event logs with Regex

lukejadamec — Tue, 04 Mar 2014 15:51:01 GMT

Sorry for the delay. I updated the answer. Be careful that you are not dropping 100% of the 4624 events.

Re: Filtering windows security event logs with Regex

andrewdidone — Thu, 06 Mar 2014 23:59:13 GMT

Hey, thanks for taking the time on this. Unfortunately i am still seeing the undesired events. Please see raw data below I believe whats causing an issue is the fact there are 2 Account Name fields in the event. Is there a way to do "Account Name: = "-" AND "machine$"

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

New Logon:
Security ID: AUSODOM\0613-09015$
Account Name: 0613-09015$
Account Domain: AUSODOM
Logon ID: 0x157ad97b2
Logon GUID: {98EF322F-
Key Length: 0

Thanks,
Andrew

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 07 Mar 2014 00:57:12 GMT

Yes.... But it will take a few minutes.

Re: Filtering windows security event logs with Regex

lukejadamec — Fri, 07 Mar 2014 01:01:03 GMT

Try this in a search first:

EventCode=4624 | regex "(?msi)^EventCode=4624\D.*Account\s+Name:\s+-.*Account\s+Name:\s+[a-z0-9-]+[\$]"

It should find all of the events you want to drop.

Re: Filtering windows security event logs with Regex

hkust — Fri, 13 Mar 2015 09:21:27 GMT

Can you share the transforms.conf on removing the "Message" and "Body" in order to reduce indexing size? Thanks.

Re: Filtering windows security event logs with Regex

sys1pmp — Thu, 07 May 2015 15:37:42 GMT

Can you explain how can I just filter out Security_ID="NULL SID". Need help to find exact regex

Re: Filtering windows security event logs with Regex

leonsanm — Wed, 02 Sep 2015 23:28:59 GMT

this works great in the search, but I tried this in my transforms and I still get the events... 😞