Can I truncate my Windows logs?

[pfabrizi]

I am being asked if we can remove some the extraneous data from our Windows logs at indexing time.
I remember in class the instructor talked about this, but I can't remember?

[spayneort]

You can use SECMD in props.conf on the indexer/heavy forwarder to remove some of the extra text in the logs. Here are some examples.

https://answers.splunk.com/answers/44865/remove-out-section-of-log.html
https://answers.splunk.com/answers/4752/disabling-or-removing-extra-description-text-in-windows-2008...
https://gist.github.com/automine/5c8ef5b50e1df38249dfba01a70f2875

[WinEventLog:Security]
#Returns most of the space savings XML would provide
SEDCMD-clean0-null_sids = s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(\:)(\s+NULL SID)$/\1/g s/(?m)(ID\:)(\s+0x0)$/\1/g
SEDCMD-clean1-summary = s/This event is generated[\S\s\r\n]+$//g
SEDCMD-clean2-cert_summary = s/Certificate information is only[\S\s\r\n]+$//g
SEDCMD-clean3-blank_ipv6 = s/::ffff://g
SEDCMD-clean4-token_elevation_summary = s/Token Elevation Type indicates[\S\s\r\n]+$//g
SEDCMD-clean5-network_share_summary = s/(?ms)(A network share object was checked to see whether.*$)//g
SEDCMD-clean6-authentication_summary = s/(?ms)(The computer attempted to validate the credentials.*$)//g
SEDCMD-clean7-local_ipv6 = s/(?ms)(::1)//g

# Removed due to issue with Windows Filtering Platform events
# SEDCMD-clean8-firewall_summary = s/(?ms)(The Windows Filtering Platform has permitted.*$)//g

[nickhills]

You sure can:
Whitelisting and blacklisting is what you are looking for!

http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Inputsconf#Event_Log_whitelist_and_blacklist...

[mayurr98]

hey There is one way to i.e. discard specific events and keep the rest using props.conf and transforms.conf

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_e...

As from the doc above follow Steps:

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

1. In props.conf, set the TRANSFORMS-null attribute:

   [source::/var/log/messages]
   TRANSFORMS-null= setnull
   

2. Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

   [setnull]
   REGEX = [sshd]
   DEST_KEY = queue
   FORMAT = nullQueue

3.Restart Splunk Enterprise.

Let me know if this helps you!

[nickhills]

Whilst this will work, this is not the most effective way to do this for windows logs - The windows event log input has a specific mechanism for dropping the large number of events windows produces, which prevents them being forwarded from the client - its far more efficient to drop them on the UF, than after they have crossed the network.