Getting Data In

remove out section of log

Path Finder

We just installed the universal forwarder on our domain controllers and are getting events just fine, however in most all of the Kerberos related events there is this junk text Windows puts in:

This event is generated when a logon session is created. It is generated on the computer that was accessed....

There are about 9 more sentences in there that are the same standard text across all the logs. Can this be removed either on the forwarder before indexing or on the indexer itself?

Tags (2)
1 Solution

Influencer

The SEDCMD-* option props.conf can be used to remove unwanted chunks of logentries.

http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Anonymizedatausingconfigurationfiles

For example:

[WinEventLog:Security]
SEDCMD-strip-detail-msg = s/(?ims)\s+^This event is generated .+//g

This has to be done at the event-processing instance. So if you're using an universal- or a lightweight forwarder, then you need to configure this on the indexer. In case of a heavy forwarder, you can strip the description at the source directly.

View solution in original post

Motivator

I was able to remove all of these verbose endings with just two SEDCMD statements:

SEDCMD-winsec1 = s/This event is generated.*[\r\n]+(.*[\r\n])+(.*$)?//
SEDCMD-winsec2 = s/Token Elevation Type indicates.*[\r\n]+(.*[\r\n])+(.*$)?//

The first one replaces your -4624, -4625, -4634, and -4769 statements, because in this format it does not matter how many lines follow the matched text.

Super Champion

good idea.

0 Karma

Explorer

Here's what we put in place in our environment - thanks to the syntax I found here. 🙂 The numbers in the line refer to the event code.

SEDCMD-strip-detail-4688 = s/(?ims)Token Elevation Type indicates .*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4624 = s/(?ims)This event is generated when.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4625 = s/(?ims)This event is generated when.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4634 = s/(?ims)This event is generated when.*//g
SEDCMD-strip-detail-4769 = s/(?ims)This event is generated every time.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g

If you want to find the extra lines in your data a search for "This event is generated" in the windows logs will point towards most of entries with extra lines.

0 Karma

Influencer

The SEDCMD-* option props.conf can be used to remove unwanted chunks of logentries.

http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Anonymizedatausingconfigurationfiles

For example:

[WinEventLog:Security]
SEDCMD-strip-detail-msg = s/(?ims)\s+^This event is generated .+//g

This has to be done at the event-processing instance. So if you're using an universal- or a lightweight forwarder, then you need to configure this on the indexer. In case of a heavy forwarder, you can strip the description at the source directly.

View solution in original post

Path Finder

It looks like your SEDCMD was correct. For some reason, it didn't like the source or sourcetype fields. When I replace it with the actual hostname, it worked just fine. Thanks!

0 Karma

Influencer

Quote from docs:

Currently the following subset of
  sed commands is supported:
        * replace (s) and character substitution (y).

See http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

0 Karma

Influencer

You should be able to simply use the example that is supplied in my answer. Not sure what your expression should actually do, but I think Splunk only supports replace and substituion mode (ie. s/some/thing/ y/some/thing/)

0 Karma

Path Finder

Yes, this is what I'm looking for except I can't get it to work. I've tried your answer and also have tried:

[WinEventLog:Security]
SEDCMD-strip-detail-msg = /This event is generated when a logon session is created/,$d

sed '/This event is generated when a logon session is created/,$d' works just fine through a linux command, any recommendations?

0 Karma

Splunk Employee
Splunk Employee

The way to reduce your indexed volume and to get rid of useless events, is to setup filtering on the indexer.

Usually based on sourcetype, and a regex mach on the events.

See example in the docs http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Filter_event_data_and_s...