Getting Data In

Disabling or removing extra description text in Windows 2008 event logs?

Super Champion

I just recently started using Windows 2008 and when I got splunk setup and forwarding thge Windows event logs and I noticed that a large number of the events now have a sentence or two about the event itself at the end of the event. This is nice when you first run across a new type of event, but it clutters up splunk with unnecessary fluff (If I wanted this info within splunk, I would prefer to create a splunk lookup table. It seems that the same message is always used for the same "EventCode"). Sometimes the extra text is quite long.

Does anybody know if this is a "feature" that can be turned off? Or, has anyone come up with a reasonable way of stripping this kind of boiler-plate text?

I'm thinking that this could greatly impact splunk's overall licensing volume as we deploy and start more heavily using Windows 2008 servers.


Here are some (truncated) samples:

07/16/10 08:11:34 AM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4648
EventType=0
Type=Information
ComputerName=WIN64.domain.com
TaskCategory=Logon
OpCode=Info
RecordNumber=5973
Keywords=Audit Success
Message=A logon was attempted using explicit credentials.
....
This event is generated when a process attempts to log on an account by explicitly specifying that account’s credentials.  This most commonly occurs in batch-type configurations such as scheduled tasks, or when using the RUNAS command.

For an even more absurd example, take a look at the following event:

07/16/10 11:18:30 AM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4624
EventType=0
Type=Information
ComputerName=HIDDEN
TaskCategory=Logon
OpCode=Info
RecordNumber=6039
Keywords=Audit Success
Message=An account was successfully logged on.
....
This event is generated when a logon session is created. It is generated on the computer that was accessed.
The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.
The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).
The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.
The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.
The authentication information fields provide detailed information about this specific logon request.
    - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
    - Transited services indicate which intermediate services have participated in this logon request.
    - Package name indicates which sub-protocol was used among the NTLM protocols.
    - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

In the last event there is more than 1k of static text, and many many terms I don't necessarily want indexed. That can add up quickly.

I know that a transformer could be used to clean this up, but I don't want to strip off a legitimate message by accident. In the two examples shown, they both start with "This event is generated when", but that's not true for all the the messages I've seen so far. So without a very large sample it seems difficult to safely come up with a regex that only removes the extra verbose message.

Splunk Employee
Splunk Employee

I had events coming from WEF in the required xml format. The Message field was massive and redundant.

In the 6.0 version of the Windows there are example sedcmd commands for removing text. I based the one below on one of those.
This was put in the TA's local directory and it had the desired effect of reducing the size of these events by ~78 % counting bytes.

[XmlWinEventLog]
SEDCMD-clean_msg_text_from_winsecurity_events = s/[\S\s\r\n]+$/-<\/Message><\/RenderingInfo>/g

I elected to replace all the text with - just in case that would be of use. I kept the Renderinginfo tag so that the Message field would be extracted.

Here is a example of the reduction.

foobar> ls -ldb Ev*
-rw-r--r--@ 1foobar staff 2764 Oct 20 17:25 EventCode4642-Nottrimmed.xml
-rw-r--r--@ 1 footer staff 602 Oct 20 17:18 EventCode4642-Trimmed.xml
foobar

0 Karma

Splunk Employee
Splunk Employee

Simple fix for removing the repeating detailed description from the message field but leave details:

in props.conf

# message shortener for windows event security
# removes text from message field starting with: This event is generated

[WinEventLog:Security]
TRANSFORM-windows_events = win_event_shortener

in transforms.conf

[win_event_shortener]
DEST_KEY = _raw
REGEX = ((.*+[\v])+)(?=This event is generated)
FORMAT = $1

Path Finder

FYI, the TRANSFORM-windows_events should be TRANSFORMS-windows_events for those that may be confused.

Motivator

Is this more efficient than the simple replace in the props.conf shown above it? I also wonder if it would be possible limiting application of the substitute or regex to events with EventCode of 4624 or 4634. This as it is would be tested against all incoming events Security events in this index.

0 Karma

Explorer

We're testing this SEDCMD configuration and it seems to work well for truncating the message text for event IDs 4624, 4634, and a few others which are the high-volume events in an Active Directory environment:

# Install on Splunk indexer in $SPLUNK_HOME/etc/system/local/props.conf
[source::WinEventLog:Security]
SEDCMD-win = s/(?mis)(Token Elevation Type indicates|This event is generated).*$//g

Strictly speaking the trailing "g" probably isn't necessary. We did run into one problem which is that agents must be configured as light forwarders for the parsing to happen at the indexer. This would need to be deployed to non-lightweight forwarders as well, I believe. Since we deployed this only two days ago and had to upgrade a bunch of agents in the process, I'm still in waiting mode to see what volume reduction this generates.

-- James

Motivator

Did this work? I tried it and it doesn't seem to be working for me.

0 Karma

SplunkTrust
SplunkTrust

Yes it does. Although I'm applying it to the WinEventLog:Security sourcetype rather than the source specified above.
It needs to go wherever the parsing stage is happening (Indexers in my case)

0 Karma

Motivator

My research and testing suggests that this feature cannot be turned off. Fortunately it appears only a limited number of events have this verbose text added to them, at least at this time. 13 events in total:

4612: Internal resources allocated for the queuing of audit messages have been exhausted, leading to the loss of some audits.
4616: The system time was changed.
4618: A monitored security event pattern has occurred
4624: An account was successfully logged on
4625: An account failed to log on
4634: An account was logged off
4647: User initiated logoff
4648: A logon was attempted using explicit credentials
4769: A Kerberos service ticket was requested
4778: A session was reconnected to a Window Station
4779: A session was disconnected from a Window Station
4780: The ACL was set on accounts which are members of administrators groups
4908: Special Groups Logon table modified

Out of these, only 4624, 4625, 4634, 4647, 4648, 4769 and possibly 4778 and 4779 are common enough to worry about stripping the description text. Based on this limited number of event IDs it might be best to just use transforms or sedcmd...

Splunk Employee
Splunk Employee

For the record, I am noting the obvious Splunk-based solution would be to use SEDCMD or a TRANSFORM to just delete it from the incoming data.

0 Karma

Motivator

Came across this today: http://technet.microsoft.com/en-us/library/ee126097(WS.10).aspx#BKMK_Controlling

Event Viewer and Resulting Internet Communication in Windows 7 and Windows Server 2008 R2 - Procedures for preventing the flow of information to and from the Internet through Event Viewer

Sounds like an interesting lead.

0 Karma

Super Champion

I don't think this helps. This seems to be talking about the Event Viewer looking up event information online, which doesn't seem to be related to the extra verbose messages at the bottom of the windows events that I'm trying to filter out.

0 Karma

Motivator

We haven't migrated to 2k8 yet, so I haven't had the pleasure of indexing 2k8 logs, but looking at group policy documentation for 2k8 R2 the following sticks out:

Machine, wdi.admx, Diagnostics: Configure scenario execution level.

Determines the execution level for Diagnostic Policy Service (DPS) scenarios.\n\n\nIf you enable this policy setting, you must select an execution level from the dropdown menu. If you select problem detection and troubleshooting only, the DPS will detect problems and attempt to determine their root causes. These root causes will be logged to the event log when detected, but no corrective action will be taken. [...] The DPS can be configured with the Services snap-in to the Microsoft Management Console.

What execution level do you have the Diagnostic Policy Service (DPS) set for? Could it be that it's configured for detection and troubleshooting, and causes it to log the extra data?

[edit] FYI 2k8 r2 group policies: http://www.microsoft.com/downloads/details.aspx?FamilyID=18c90c80-8b0a-4906-a4f5-ff24cc2030fb&displa...

0 Karma

Motivator

Hmm, doing some testing and this definitely did not help.

0 Karma

Motivator

Hmm we don't have win 2k8 deployed yet, so I am not sure, but it just seems like there should be GPO to customize the logs. Haven't been able to find info on the google yet...

0 Karma