topic Can I truncate my Windows logs? in Getting Data In

Can I truncate my Windows logs?

pfabrizi — Fri, 12 Jan 2018 16:47:08 GMT

I am being asked if we can remove some the extraneous data from our Windows logs at indexing time.
I remember in class the instructor talked about this, but I can't remember?

Re: Can I truncate my Windows logs?

mayurr98 — Fri, 12 Jan 2018 16:55:24 GMT

hey There is one way to i.e. discard specific events and keep the rest using props.conf and transforms.conf

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

As from the doc above follow Steps:

This example discards all sshd events in /var/log/messages by sending them to nullQueue:

In props.conf, set the TRANSFORMS-null attribute:

[source::/var/log/messages]
TRANSFORMS-null= setnull

Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue

3.Restart Splunk Enterprise.

Let me know if this helps you!

Re: Can I truncate my Windows logs?

nickhills — Fri, 12 Jan 2018 16:58:16 GMT

You sure can:
Whitelisting and blacklisting is what you are looking for!

http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/Inputsconf#Event_Log_whitelist_and_blacklist_formats

Re: Can I truncate my Windows logs?

nickhills — Fri, 12 Jan 2018 17:01:12 GMT

Whilst this will work, this is not the most effective way to do this for windows logs - The windows event log input has a specific mechanism for dropping the large number of events windows produces, which prevents them being forwarded from the client - its far more efficient to drop them on the UF, than after they have crossed the network.

Re: Can I truncate my Windows logs?

spayneort — Sat, 13 Jan 2018 16:07:00 GMT

You can use SECMD in props.conf on the indexer/heavy forwarder to remove some of the extra text in the logs. Here are some examples.

https://answers.splunk.com/answers/44865/remove-out-section-of-log.html
https://answers.splunk.com/answers/4752/disabling-or-removing-extra-description-text-in-windows-2008-event-logs.html
https://gist.github.com/automine/5c8ef5b50e1df38249dfba01a70f2875

[WinEventLog:Security]
#Returns most of the space savings XML would provide
SEDCMD-clean0-null_sids = s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(\:)(\s+NULL SID)$/\1/g s/(?m)(ID\:)(\s+0x0)$/\1/g
SEDCMD-clean1-summary = s/This event is generated[\S\s\r\n]+$//g
SEDCMD-clean2-cert_summary = s/Certificate information is only[\S\s\r\n]+$//g
SEDCMD-clean3-blank_ipv6 = s/::ffff://g
SEDCMD-clean4-token_elevation_summary = s/Token Elevation Type indicates[\S\s\r\n]+$//g
SEDCMD-clean5-network_share_summary = s/(?ms)(A network share object was checked to see whether.*$)//g
SEDCMD-clean6-authentication_summary = s/(?ms)(The computer attempted to validate the credentials.*$)//g
SEDCMD-clean7-local_ipv6 = s/(?ms)(::1)//g

# Removed due to issue with Windows Filtering Platform events
# SEDCMD-clean8-firewall_summary = s/(?ms)(The Windows Filtering Platform has permitted.*$)//g