I just recently started using Windows 2008 and when I got splunk setup and forwarding thge Windows event logs and I noticed that a large number of the events now have a sentence or two about the event itself at the end of the event. This is nice when you first run across a new type of event, but it clutters up splunk with unnecessary fluff (If I wanted this info within splunk, I would prefer to create a splunk lookup table. It seems that the same message is always used for the same "EventCode"). Sometimes the extra text is quite long.
Does anybody know if this is a "feature" that can be turned off? Or, has anyone come up with a reasonable way of stripping this kind of boiler-plate text?
I'm thinking that this could greatly impact splunk's overall licensing volume as we deploy and start more heavily using Windows 2008 servers.
Here are some (truncated) samples:
07/16/10 08:11:34 AM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4648 EventType=0 Type=Information ComputerName=WIN64.domain.com TaskCategory=Logon OpCode=Info RecordNumber=5973 Keywords=Audit Success Message=A logon was attempted using explicit credentials. .... This event is generated when a process attempts to log on an account by explicitly specifying that account’s credentials. This most commonly occurs in batch-type configurations such as scheduled tasks, or when using the RUNAS command.
For an even more absurd example, take a look at the following event:
07/16/10 11:18:30 AM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4624 EventType=0 Type=Information ComputerName=HIDDEN TaskCategory=Logon OpCode=Info RecordNumber=6039 Keywords=Audit Success Message=An account was successfully logged on. .... This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
In the last event there is more than 1k of static text, and many many terms I don't necessarily want indexed. That can add up quickly.
I know that a transformer could be used to clean this up, but I don't want to strip off a legitimate message by accident. In the two examples shown, they both start with "
This event is generated when", but that's not true for all the the messages I've seen so far. So without a very large sample it seems difficult to safely come up with a regex that only removes the extra verbose message.
I had events coming from WEF in the required xml format. The Message field was massive and redundant.
In the 6.0 version of the Windows there are example sedcmd commands for removing text. I based the one below on one of those.
This was put in the TA's local directory and it had the desired effect of reducing the size of these events by ~78 % counting bytes.
SEDCMD-clean_msg_text_from_winsecurity_events = s/[\S\s\r\n]+$/-<\/Message><\/RenderingInfo>/g
I elected to replace all the text with - just in case that would be of use. I kept the Renderinginfo tag so that the Message field would be extracted.
Here is a example of the reduction.
foobar> ls -ldb Ev*
-rw-r--r--@ 1foobar staff 2764 Oct 20 17:25 EventCode4642-Nottrimmed.xml
-rw-r--r--@ 1 footer staff 602 Oct 20 17:18 EventCode4642-Trimmed.xml
Simple fix for removing the repeating detailed description from the message field but leave details:
in props.conf # message shortener for windows event security # removes text from message field starting with: This event is generated [WinEventLog:Security] TRANSFORM-windows_events = win_event_shortener in transforms.conf [win_event_shortener] DEST_KEY = _raw REGEX = ((.*+[\v])+)(?=This event is generated) FORMAT = $1
Is this more efficient than the simple replace in the props.conf shown above it? I also wonder if it would be possible limiting application of the substitute or regex to events with EventCode of 4624 or 4634. This as it is would be tested against all incoming events Security events in this index.
We're testing this SEDCMD configuration and it seems to work well for truncating the message text for event IDs 4624, 4634, and a few others which are the high-volume events in an Active Directory environment:
# Install on Splunk indexer in $SPLUNK_HOME/etc/system/local/props.conf [source::WinEventLog:Security] SEDCMD-win = s/(?mis)(Token Elevation Type indicates|This event is generated).*$//g
Strictly speaking the trailing "g" probably isn't necessary. We did run into one problem which is that agents must be configured as light forwarders for the parsing to happen at the indexer. This would need to be deployed to non-lightweight forwarders as well, I believe. Since we deployed this only two days ago and had to upgrade a bunch of agents in the process, I'm still in waiting mode to see what volume reduction this generates.
Yes it does. Although I'm applying it to the WinEventLog:Security sourcetype rather than the source specified above.
It needs to go wherever the parsing stage is happening (Indexers in my case)
My research and testing suggests that this feature cannot be turned off. Fortunately it appears only a limited number of events have this verbose text added to them, at least at this time. 13 events in total:
4612: Internal resources allocated for the queuing of audit messages have been exhausted, leading to the loss of some audits. 4616: The system time was changed. 4618: A monitored security event pattern has occurred 4624: An account was successfully logged on 4625: An account failed to log on 4634: An account was logged off 4647: User initiated logoff 4648: A logon was attempted using explicit credentials 4769: A Kerberos service ticket was requested 4778: A session was reconnected to a Window Station 4779: A session was disconnected from a Window Station 4780: The ACL was set on accounts which are members of administrators groups 4908: Special Groups Logon table modified
Out of these, only 4624, 4625, 4634, 4647, 4648, 4769 and possibly 4778 and 4779 are common enough to worry about stripping the description text. Based on this limited number of event IDs it might be best to just use transforms or sedcmd...
Came across this today: http://technet.microsoft.com/en-us/library/ee126097(WS.10).aspx#BKMK_Controlling
Event Viewer and Resulting Internet Communication in Windows 7 and Windows Server 2008 R2 - Procedures for preventing the flow of information to and from the Internet through Event Viewer
Sounds like an interesting lead.
I don't think this helps. This seems to be talking about the Event Viewer looking up event information online, which doesn't seem to be related to the extra verbose messages at the bottom of the windows events that I'm trying to filter out.
We haven't migrated to 2k8 yet, so I haven't had the pleasure of indexing 2k8 logs, but looking at group policy documentation for 2k8 R2 the following sticks out:
Machine, wdi.admx, Diagnostics: Configure scenario execution level.
Determines the execution level for Diagnostic Policy Service (DPS) scenarios.\n\n\nIf you enable this policy setting, you must select an execution level from the dropdown menu. If you select problem detection and troubleshooting only, the DPS will detect problems and attempt to determine their root causes. These root causes will be logged to the event log when detected, but no corrective action will be taken. [...] The DPS can be configured with the Services snap-in to the Microsoft Management Console.
What execution level do you have the Diagnostic Policy Service (DPS) set for? Could it be that it's configured for detection and troubleshooting, and causes it to log the extra data?
 FYI 2k8 r2 group policies: http://www.microsoft.com/downloads/details.aspx?FamilyID=18c90c80-8b0a-4906-a4f5-ff24cc2030fb&displa...